From our data · AI search measurement
You asked ChatGPT once and it named you. That is not a measurement.
Everyone starts the same way. Type the category question into ChatGPT, see whether the brand comes back, screenshot it if it does. The result feels solid. It is closer to a coin toss.
By Priyadharshini Krishnan, Co-Founder & Chief AI Officer · Updated 16 September 2026 · 7 minute read
A language model generates its answer, and generating involves sampling. Ask the same question twice and the wording shifts. Ask it ten times and the brands named can shift too. So one run tells you that you can appear. Whether you actually do is a different claim. The honest unit is a rate: how many of ten runs of the same question named you. Run each question several times, write down the proportion, and watch that proportion over weeks. Three out of ten is a number you can act on and argue about. One favorable answer is an anecdote. Nothing here costs money, only patience, and it is most of the difference between a figure that survives a budget meeting and one that does not.
Why does the same question give different answers?
The model is predicting likely next words and sampling from those predictions. Two runs are two draws from a distribution. Nothing was looked up, so nothing has to match.
Turning the temperature down to zero narrows the spread without closing it. Variation has been observed at temperature zero, because the hardware underneath performs floating point arithmetic in an order that is not guaranteed to repeat between runs. For anyone measuring a brand, the useful version of that finding is short: one answer is not the answer.
Then there is a second kind of movement, and it has nothing to do with sampling. The engines themselves change. Semrush took weekly snapshots across thirteen weeks and watched Reddit go from roughly 60% of ChatGPT citations in early August to roughly 10% by mid September. Wikipedia fell from 55% to under 20% over the same weeks. Neither site did anything. ChatGPT changed where it looked.
What should you count instead?
A rate, across a fixed set of questions, tracked over time. Three decisions make it worth trusting.
- Repeat every question. The April 2026 paper on this is called "Don't Measure Once", which saves anyone the trouble of reading further: single measurements are not statistically stable and repeated ones are. Practitioners tend to settle between five and ten runs.
- Report the proportion. "Named in 4 of 10 runs" can go up next month. "ChatGPT mentioned us" cannot go anywhere.
- Hold the question set still. Swap questions in and out between rounds and a change in the score might only be a change in what you asked. Fix the set, change it deliberately, note the date you did.
The comparison that does the work
Your rate on its own is oddly hard to read. Is 30% good? Nobody knows. The same rate beside two named competitors, on the same questions in the same runs, needs no explanation at all: you are in 3 of 10, they are in 8 of 10, and this is a question your buyers ask before they buy. Finance directors understand that chart without a briefing.
Who decides whether the brand was mentioned?
Most teams skip past this, and it quietly ruins the number. Somebody has to read each answer and rule on it. Were you named? Named approvingly? Alluded to without being named, which happens more than you would expect? Ask a person to do that across four hundred answers and their standard drifts by Thursday. Ask a model and you have a new question, which is: which model.
Our rule is that the model doing the judging is never the model that wrote the answer, and never from the same vendor. A system that produces an answer and then rules on whether that answer flattered you is marking its own homework. Enforcing this costs nothing and removes a whole category of doubt from every number that comes out.
One good answer proves you are possible. It does not prove you are present. Only one of those claims survives contact with a budget review.
What to do this week
- Write ten questions a buyer would really type, in the words your buyers use, which are seldom your category's.
- Run each five times in one assistant and count how many name you. An afternoon, and you have a baseline.
- Do the same for two competitors, same questions, same session.
- Come back in a fortnight. The movement is the signal. The first number, on its own, is not.
If you are an agency
This is the difference between a client report that holds up and one that falls apart in the room. Screenshot a flattering answer and it is worth nothing the moment the client runs the question themselves and sees something else, which they will do on the train home. A rate across repeated runs, with competitors measured in the same runs, is defensible and shows movement month on month. It also sets an expectation you can actually meet, because you are reporting a proportion you can stand behind.
How we do this ourselves
This is what our AI search visibility tracking is for. Questions are saved and run on a schedule, every run is recorded so a rate can be calculated, and the model that decides whether your brand appeared is always resolved to a different vendor than the one that answered. Google AI Overview results are appended, so the history stays intact and a trend is readable.