Auto Alpha AdvisoryIf you pay for AI visibility work, judge it on a fixed set of buyer questions, against a noise band you set before you look, and over a quarter rather than a week. A single headline number, checked weekly, will mostly show you noise.
I can say that with some confidence because I publish an instrument that measured it. The SA AI Visibility Index tracked the same 42 brands in property, legal and accounting twice: first in June (property) and July (legal and accounting), then again on 25 September 2026. Edition 2 was published on 9 October 2026, from 1,114 answers. No tracked estate agency, law firm or accounting firm moved by 5 points or more on both of its sample rules. Two platforms did.
My earlier study of 832 AI answers looked at what gets a business cited. This one asks the follow-up question that matters to anyone paying for this work: once you're being tracked, how would you know whether anything changed?
What changed between the two sweeps: two platforms, no tracked firm
Edition 2 put the same 108 buyer questions to four AI models, the API versions of the models behind ChatGPT, Claude, Gemini and Perplexity, three runs per question. The rules were fixed and published on 3 September 2026, before the second sweep. The one that matters most here: a share of mention moving by fewer than 5 percentage points counts as no change.
Against that bar, legal showed no movement at all. In property, 13 of the 14 tracked brands didn't move, and Property24's share of mention fell from 15% to 10%, a drop of 5.7 points. In accounting, 10 of the 14 didn't move, and Xero's share fell from 16% to 9%, a drop of 7.1 points. Three accounting firms crossed the line on one sample rule and not the other, so the Index does not report them as movement.
That leaves two brands out of 42, both platforms, named in fewer answers than before. The SA AI Visibility Index can't tell you why, and the Index's method note says so plainly: "a number that moves is not proof that anything a brand did, or anything we advised, caused it." It records what four AI models said on two dates. It says nothing about causes.
The Index also missed its own dates, and it reports that too. The sweep was pre-registered for 15 September 2026 and ran on 25 September, ten days late. Publication was pre-registered for 6 October and happened on 9 October, three days late. A rule set in advance is only worth something if you also own up to where you broke it.
A two-point rise in a monthly report is noise
Across roughly three months, a fixed 5-point rule caught movement in two of 42 brands. On that evidence, I'd say most of what moves in AI answers from one month to the next is noise, and a report that celebrates a two-point rise is reporting noise, however good the chart looks.
As a CA(SA) I think of this as materiality. You decide how big a variance has to be before it matters, and you decide it before you open the numbers, because afterwards you'll be tempted to call any move you like significant. AI answers need the same discipline because they vary between runs. That's why the Index asks every question three times instead of once.
Which answers you count changes which firms appear
Edition 2 counts an answer only if the model searched the web for it. That sounds like a technicality, but it may be the biggest judgement call in the whole method.
A model that has a search tool decides for itself whether to use it. OpenAI's web search guide says the model "can choose to search the web or not based on the content of the input prompt", and Anthropic's web search documentation says "Claude determines when to search based on the prompt." In Edition 2, the ChatGPT arm declined to search in 151 of its 324 answers, and Claude in 31.
Accounting shows what that does. ChatGPT searched the web on every city question and named local practices from its sources, none of them among the twelve tracked firms. On the national questions it mostly didn't search. It declined on 61 of its 81, on questions such as "who are the best accounting firms in South Africa", and answered those from memory. In those 61 answers it named BDO in 24, Deloitte in 19 and PwC in 17.
So whether you count the answers a model gave without searching changes which firms appear. That's why the Index reports a move only when it holds on both rules. This is what it observed in one vertical, with one instrument, on two dates, and I wouldn't stretch it into a general law. It does give you a fair question for anyone selling you AI visibility reporting: which answers do you count, and does the result survive if you count them the other way?
How to measure AI visibility without paying for a tool
You can run a rough version of this yourself in an afternoon once a quarter. Start by not typing your company name into ChatGPT. One answer to a question about yourself tells you nothing about whether a buyer who has never heard of you will find you.
- Write down ten questions a buyer would actually ask. "Which estate agents in Cape Town are good at selling sectional title flats?" is a buyer question. Your firm's name is not.
- Run each question several times in each of the four assistants: ChatGPT, Claude, Gemini and Perplexity. Use a fresh chat for every run. Answers vary from run to run, so one run proves very little.
- Log who is named in a spreadsheet, with the date, the assistant, the question, the run number and every business in the answer.
- Write your noise band down before you look at the results, and make it generous. Ten questions run three times in one assistant gives you 30 answers, so one answer either way moves your share by more than three points.
- Run the same questions the same way next quarter and compare the two.
The consumer apps won't behave exactly like the API versions the Index uses. That's fine, as long as you compare like with like: the same questions, the same assistants, the same number of runs.
Search Console tells you how often you appear, not for which questions
Google now gives you part of this for free. Search Console's Generative AI performance report counts impressions in AI Overviews and AI Mode, and Google's help page for it says: "As of August 31, 2026, we've rolled out these insights to all websites worldwide." An impression is a time a link to your site was shown in one of those features.
My own report showed my site 318 times in three months. That looked good until I opened the breakdown: almost four in five of those showings were for one post about a prompting framework. Students find that post. The business owners I work for mostly don't.
The report splits its data by page, country, device and date, and Google's help page lists no query view. It can tell you which page was shown, but not which question it answered. A useful report on AI visibility has to say which questions you appear for, not only how often, and that takes a fixed panel of questions. Back to the spreadsheet.
What a monthly report should carry instead of one number
Aleyda Solis's three-layer framework splits AI search measurement into presence, readiness and business impact, and I think it's the right split. Her BrightonSEO deck puts the layers as questions: "Are you actually appearing, and how?", "Are you structurally prepared to be surfaced?" and "Are visibility and readiness translating into value?"
The 3-Layer Report in my Monthly Advisory Retainer, from its Growth tier up, is built on the same split. Presence is who four AI models name for your buyers' questions. Readiness is whether your site can be read and cited, scored across the nine domains my audit engine checks. Business impact is what actually arrived, tied back to leads and revenue. From the same tier, the retainer also tracks your AI visibility across four AI models.
It reports monthly, and a monthly report isn't wrong. Read each month against the noise band and judge the trend over a quarter. A move inside the band is no change, whichever way the arrow points.
If you're already paying for AI visibility work, ask your provider three things: which questions they track, which answers they count, and what size of move they set as no change before they looked. If they can't answer, you can't tell their results from noise. The retainer page sets out the tiers and what they cost, and the first call is 15 minutes.