From our data · Social listening
Listening tools tell you volume. They do not tell you what people said.
A mention count can be technically correct and completely useless at the same time. Four ways that happens, found by reading every mention behind a single number.
20 August 2026 · 9 minute read
A mention count is only as good as the relevance test behind it, and most relevance tests cannot explain themselves. We audited our own by reading the actual mentions behind a single screen, and found four distinct failures: financial coverage of a competitor counted as brand conversation, theme keywords sent to news APIs as literal search terms, brand-name matches treated as proof of relevance, and a timeout quietly excluding every social source so the tool only ever heard newspapers. Each was individually reasonable and collectively fatal. The lesson generalises: if your listening tool cannot tell you why a mention is in the count, the count is not evidence.
Every example below is anonymised. We do not publish findings about named brands, including our own customers.
Failure one: market coverage is not conversation
A regional ice cream brand had its top topics returned as quarterly earnings, dividends and record dates. Every one of those articles genuinely mentioned the brand. The relevance filter was working exactly as designed and it was right to keep them by its own logic.
Not one of them was anybody talking about ice cream. Financial coverage of a listed company in your category is legitimately about that company and tells you nothing about how consumers feel. It is a different question with a different answer, and mixing it into a sentiment score corrupts both.
The fix was to set financial and corporate coverage aside on the news sources specifically, by publication section and phrasing, and to record why each item was set aside. Consumer sources keep everything, because someone posting on a forum about a competitor's results is genuinely talking.
Failure two: a theme is not a search term
Keywords come in kinds, and only some of them are searchable. A brand name is a sensible thing to send to a news API. A theme like "organic farming practices" is not; it is a description of a conversation you want to find, not a string that will appear in a headline about you.
For one grocery brand, a single common word and its hashtag brought in 275 of 436 total mentions. The results were about organic farming, organic chemistry and organic growth in earnings reports. Two thirds of the brand's entire mention volume was noise generated by its own keyword configuration.
Names now go to news sources. Themes go to the consumer sources, where searching a theme is exactly how you find a conversation you are not named in.
Failure three: carrying your name is not proof
A brand whose name is also an animal was accumulating wildlife and nature stories: zoo reports, sightings, documentaries. Fifty-three of them. The filter kept every one because each contained the brand name, and containing the brand name had been treated as sufficient evidence of relevance.
It is not, and the shortcut is seductive precisely because it works for most brands. Any brand whose name is also a common noun breaks it immediately. After the fix, that brand had one mention in the period: a genuine story about a competitor.
What made the difference
The relevance check was being asked to judge whether a post was about a brand while being told only the brand name and a category label. It now gets the brand description and the competitor list as well, so it is judging against what the brand actually sells rather than against a word.
Failure four: the tool could only hear newspapers
This was the largest and the least visible. Every mention in the database, across every brand, had come from a news wire. Not one from Instagram, X, Reddit, YouTube or the app stores. It looked like a set of brands with no social presence.
It was a timeout. Each source had 45 seconds to respond. We measured properly: Instagram takes 53 to 66 seconds for a single keyword, X takes 44 to 66. Both were being cancelled on every run since the module shipped. A tool built to hear people had been hearing only newspapers for its entire life, and every downstream analysis was clustering the same newspaper.
Three numbers now hold together and changing one without the others breaks it again: the per-source limit, the deadline for the whole collection, and the point at which a job is presumed dead. That last one had been set shorter than the new deadline, which would have meant a run doing exactly what it was told being reported as failed.
What generalises from this
- Read the rows. All four failures were invisible in aggregate and obvious within ten minutes of reading actual mentions. Do this monthly.
- A filter must explain itself. The first question anyone asks about a filter is what it removed. A filter that cannot answer that is asking for trust it has not earned.
- Set aside, do not delete. Everything excluded should stay visible with a reason, and the user should be able to overrule it.
- Measure before you tune. The timeout was set to a number that felt reasonable. Nobody had timed the sources.
- Beware the shortcut that works for most cases. Name-matching is right for the majority of brands, which is exactly why it survived so long.
We shipped a listening tool that could only hear newspapers, and it took reading the mentions behind one screen to notice. The number looked fine the whole time. That is the part worth remembering.
If you are an agency
If you report listening numbers to clients, audit the mentions behind them before the next review, not after a client does. Pull fifty rows at random and read them. If more than a handful are not genuinely about the brand, the trend line you have been presenting is measuring your keyword configuration rather than the market. This is a two-hour exercise that prevents a very uncomfortable meeting.
How we do this ourselves
Social Listening now runs the relevance test before anything is stored, and shows you what it set aside and why so you can overrule it. The four fixes above are all in the product, and the timeout numbers are pinned by a test so they cannot silently regress. It is one of two flagship modules in Signal, alongside the AEO/GEO Monitor.
How Social Listening works