We tested 60 AI search answers. Visibility was the easy metric. Value was not.

We ran the same four commercial buying prompts five times in each of three AI search systems. Sixty anonymous answers. Our consultancy, letsrocc, appeared in none of them. It was not mentioned, shortlisted or cited once.

That result is uncomfortable, useful and easy to misuse.

The wrong conclusion is that one firm has no authority while the suppliers named most often have proved theirs. The experiment cannot support either claim. An AI visibility result is not decision-grade unless the provider discloses the prompt sample, repeatability conditions, source variance and the commercial outcome. A citation is evidence of presence in a defined sample. It is not evidence of demand.

Our commercial interest

Our commercial interest is direct. letsrocc provides senior SEO and commercial diagnostic work to ecommerce and other organisations, so we can be considered for some of the same problems as several suppliers named in the answers. A stronger visibility result could benefit us, and a weak one could expose a gap in how our work is understood.

That is why we fixed the prompts and test conditions before recording the answers, disclosed our own zero result and made no judgement about the quality of any named supplier. No named supplier was involved in selecting the prompts or recording the results. The experiment measured what three systems returned under those conditions. It didn't compare delivery quality, capability or commercial results.

We were testing repeatability, not trying to win

The experiment began with one question: how stable are brand mentions, recommendations and cited sources when the same commercial prompt is repeated under controlled conditions?

We used four prompts based on real supplier-selection situations:

• finding an SEO consultancy for a mid-sized ecommerce company operating across the UK and Germany
• finding senior specialists for a complex international ecommerce website
• finding help after organic traffic fell following a website rebuild
• finding a consultancy able to examine both technical SEO and the commercial causes of weak organic performance

For each prompt, we ran five independent conversations in Google AI Mode, Gemini and ChatGPT Search. Each run began in a new conversation with the wording unchanged. We recorded the suppliers shown, their order, whether letsrocc appeared, and whether any source or citation could be inspected.

That produced 60 comparable answers: four prompts, three systems and five runs per system. Claude was not included in this pilot, so the findings should not be read as evidence about Claude’s recommendations or sourcing behaviour.

We also ran a separate context-aware ChatGPT control. It already knew letsrocc from previous conversations. It mentioned us in four of five runs for the first prompt, but never placed us in the primary shortlist and never cited us. We excluded those runs from the comparable benchmark because prior context had changed the test condition.

That exclusion matters. A context-aware system acknowledging a familiar company is not the same as a new conversation recommending it to an unknown buyer. The first result to examine was how consistently the systems selected suppliers.

The supplier lists moved more than the scores suggest

The first clear finding was not our absence. It was the instability of the recommendations.

Different systems repeatedly selected different suppliers for the same buying question. For the website-rebuild prompt, Google AI Mode placed BLKDG and Seybold in its leading pair in four of five runs. Gemini placed Peak Ace and StudioHawk in its leading pair in four of five. ChatGPT Search favoured mindmelt and code.marketing in three of five runs. The six leading positions therefore contained six different providers. Of those six, Seybold was the only provider to appear in another system's results, once in ChatGPT Search.

The technical SEO and commercial underperformance prompt showed the same dependence on the system. Audits.com appeared in all five Google AI Mode runs, one Gemini run and no ChatGPT Search runs. Ayima appeared in three Gemini runs, one ChatGPT Search run and no Google AI Mode runs. No provider appeared across all three systems.

Even where one supplier appeared consistently, the surrounding shortlist changed considerably. A company could therefore receive a reassuring visibility score from one provider while being absent from a different system, prompt formulation or market.

The score may be calculated correctly. The decision built on it may still be wrong.

This does not make AI search monitoring useless. It means the measurement boundary must be stated. A result gathered from ten prompts in one logged-in system is evidence about those ten prompts, in that system, under those conditions. It should not be presented as a general measure of market authority.

Source transparency was also system-dependent

The answers varied not only in which firms they recommended, but in how much of the evidence behind them could be inspected.

In our captured runs, ChatGPT Search usually exposed direct provider URLs. Google AI Mode showed opaque redirect references rather than the final source destinations. Gemini showed no source URLs in the captured answers.

A visible source does not validate the claim attached to it. It does allow somebody to inspect what the system relied upon. Without that, a leadership team cannot tell whether a recommendation reflects a detailed case study, a directory profile, a list article, a supplier's own service page or repeated mentions across weak sources.

This becomes commercially relevant when a monitoring report treats a citation as an authority signal without showing the evidence behind it.

A brand can be cited for the wrong reason, against the wrong service, from a page that attracts no qualified buyer, or in an answer nobody acts upon. Counting the citation does not resolve any of those questions.

Mention, citation and recommendation are different observations

AI visibility reporting often compresses unlike events into a single score.

A brand mention means the organisation appeared by name. A citation means a page or domain was presented as a source. A recommendation means the system placed the supplier into the user's consideration set. A referral visit means somebody clicked through. A qualified enquiry means the visitor had a relevant need and enough intent to act.

Those are not interchangeable stages of one proven funnel.

Our experiment recorded mentions, shortlist positions and citations separately for that reason. letsrocc scored zero across all three in the comparable benchmark. We cannot convert that into an estimate of lost demand: we do not know how many real buyers use those prompts, how frequently they see comparable answers, whether they trust them or whether they click and buy.

The same restraint must apply when the result is positive.

If a company appears in two out of five sampled answers, it has established presence within that sample. That is not evidence that two fifths of its market saw the brand, that the mentions created traffic, or that the traffic produced revenue. Those limits apply equally to a zero result and a positive visibility score.

The strongest objection is correct

This was a small pilot conducted within one test window. Models change. Search modes change. Location, language, account state, personalisation and new web content can alter the response. Four prompts cannot represent an entire buying market.

That does not weaken the central finding. It defines it.

We are not claiming that the named suppliers are universally more visible, that letsrocc will remain absent, or that any system is more reliable. We are showing that repeated answers varied enough to make a single result an unsafe commercial conclusion.

A larger study may produce more stable estimates. It would still need to disclose the prompts, systems, dates, markets, account conditions and classification rules. Scale does not correct an opaque method.

The temptation is to solve this by adding more prompts until the number feels scientific. That can create a more precise description of the provider's own sample without establishing whether the sample represents actual buyer behaviour.

The missing denominator remains the market.

What leadership should ask before paying for AI visibility

The first question is not, "What is our score?" It is, "What decision are we trying to make?"

If the decision concerns content investment, the report should show which prompts, markets and buyer situations expose a relevant evidence gap. If the decision concerns brand authority, it should show how citations were classified and whether the sources are independently inspectable. If the decision concerns commercial return, the organisation must connect the observation to first-party behaviour.

Before accepting a visibility result, leadership should be able to see:

• the exact prompt set and why it represents the intended buyers
• the systems, model or search mode, market, language and account state
• the number of independent runs and the amount of variance between them
• the definitions of mention, citation, recommendation and referral
• the underlying answers and source destinations needed to audit the classification
• the first-party evidence connecting any visibility to visits, enquiries, opportunities, revenue or margin

This is not a demand for perfect attribution. It is a demand that the evidence match the decision.

A report can justify improving measurement without justifying more content. It can justify a bounded test without supporting a larger programme. It can show that nothing further should be done. Those are useful outcomes when the purpose is a better decision rather than a more impressive dashboard.

Our zero was more valuable than a flattering score

The experiment did not give us a claim we could put on a sales page. It gave us a clearer description of the evidence we lack.

For the four buying situations tested, our general service language was not enough to make letsrocc visible in the anonymous answers. The site described technical and commercial capability, but independently corroborated proof for those exact use cases was weaker. That is a publishing and evidence problem, not permission to manufacture generic AI content.

The practical response is to improve inspectable proof for the situations we can genuinely evidence, publish the method and its limitations, and repeat the test once those assets have been live long enough to be found. If the result changes, we will still need to distinguish visibility from commercial performance.

What I would do differently

I would refuse to pay for:

• a headline AI visibility score without the prompts, markets, systems, repeat runs, source evidence and variance behind it
• an unexplained share-of-voice number
• a content programme justified only by moving either metric

I would fund:

• representative buyer prompts tied to a specific commercial decision
• repeated tests across the systems relevant to those buyers
• retained answers and inspectable sources
• reconciliation with AI referrals, qualified enquiries, accepted opportunities, revenue and margin

If that evidence doesn’t change a decision, I would stop funding the measurement rather than keep polishing the dashboard.

That is slower than promising to make a brand visible in AI answers. It is also more defensible.

The experiment's most useful result was not 0 out of 60. It was learning how little that number can support on its own.