Are AI Citations the New Vanity Metric?
They can be, if you’re focusing on quantity without relevance or context.
AI citations are useful.
So are AI mentions, recommendations, Share of Voice and the various other measurements appearing alongside our increasingly AI-influenced search landscape.
We should measure them. We should understand whether they’re increasing or decreasing. We should know what content is being surfaced and where our businesses are appearing.
But there is a danger that, in our enthusiasm for a shiny new collection of metrics, we start treating the number itself as evidence of success.
We’ve Been Here Before
Around 15 years ago, I started working with an engineering firm and, as part of understanding its existing marketing, I took a look at the website’s SEO performance.
On paper, the numbers looked fantastic.
The website was attracting significant organic traffic. Pages were ranking well. Lots of people were finding the site through Google.
There was just one problem.
A large proportion of that traffic was arriving through a case study about work the engineering company had completed for a well-known high street food chain in the local area.
The case study had been deliberately optimised around the chain’s name and location.
And it had worked.
It ranked extremely well.
Unfortunately, the people finding it weren’t looking for engineering services.
They were looking for the restaurant.
Opening hours. Menus. Directions. Probably whether they could get a table for lunch.
Instead, they landed on an engineering case study, realised almost immediately that it wasn’t what they wanted and went straight back to the search results.
Lots of traffic. Very little relevance. And a rather impressive bounce rate to go with it.
The SEO numbers looked good.
The business value didn’t.
More importantly, the optimisation itself hadn’t failed. It had successfully achieved exactly what it had been designed to achieve.
It had just successfully achieved the wrong thing.
You can successfully improve a metric and still achieve absolutely nothing useful for the business.
Today, very few experienced SEO professionals would look at that traffic and declare the strategy a success.
Search marketing has matured considerably. We’ve learned that traffic needs context. Rankings need relevance. Keywords need intent. Leads need quality.
We didn’t stop measuring the numbers.
We got better at understanding what the numbers actually meant.
And that’s worth remembering now.
SEO has had decades to mature.
AEO, GEO, AI search optimisation and the tools we’re using to measure AI visibility are still relatively young.
New metrics are appearing. New reporting platforms are developing. Methodologies are evolving.
Importantly, not all platforms currently measure these metrics in the same way. There is a growing amount of AI visibility data available, often presented using the same or very similar terminology, but that doesn’t necessarily mean the underlying measurement is the same.
Even within a single platform, methodology can vary according to the tool being used. For example, Semrush’s AI Visibility Toolkit calculates AI Share of Voice using how often a brand is mentioned and how prominently it appears. Its Enterprise AIO product also factors topic search volume into the calculation for ChatGPT.
Source: Semrush: How to Measure AI Share of Voice
That doesn’t make the data useless.
It makes understanding the methodology important.
Experience should tell us not to make the same mistake twice.
So, Are AI Citations Vanity Metrics?
No.
Not inherently.
An AI citation can indicate that our content has been referenced as a source within an AI-generated response.
That’s useful information.
We might want to know how frequently we’re being cited, which pages are being cited, which AI platforms are citing us, how that changes over time and how our visibility compares with other businesses or sources.
Likewise, AI mentions, recommendations and Share of Voice can all help us understand how our businesses are appearing within AI-assisted discovery.
If you’re wondering about the difference between a mention, citation and recommendation, or what AI Share of Voice actually means, we’ve covered those definitions in AI Search Jargon Explained.
An AI citation becomes a vanity metric when the number is treated as evidence of marketing success without considering the relevance, context or potential business value of those citations.
The problem isn’t that we’re measuring these things.
The problem starts when the number becomes the objective.
400 AI Citations. Great. Citations for What?
Imagine The Last Hurdle is cited 500 times in AI-generated answers about the meaning of GEO.
That’s lovely.
We’ve published content explaining GEO, so being cited as a useful source would tell us that the content is being found, understood and considered relevant to that subject.
Now imagine that, during the same period, The Last Hurdle appears 50 times in relevant answers to business owners actively looking for marketing support.
500 is the bigger number.
But if our objective is to generate enquiries from businesses that need marketing support, it doesn’t automatically represent the more commercially valuable visibility.
The same applies in almost any industry.
A commercial heating company might receive hundreds of citations around questions such as:
What temperature should I set my thermostat to?
Useful content. Relevant to heating. Potentially worthwhile visibility.
But compare that with appearing less frequently when someone asks:
Who can service a commercial boiler in Milton Keynes?
Both represent AI visibility.
They don’t necessarily represent the same quality of AI visibility.
And that’s why the question after “How many citations did we get?” should be:
Citations for what?
Relevance Gives the Number Meaning
“Quality over quantity” is easy to say.
It’s also sufficiently vague to be almost useless unless we define what quality actually means.
When we’re assessing AI visibility, there are several different types of relevance worth considering.
Topic relevance: Are we appearing around the subjects, products, services and areas of expertise we actually want to be known for?
Audience relevance: Is that visibility likely to put us in front of the people we’re trying to reach?
Intent: What is the person asking the question actually trying to achieve?
Geographical relevance: Where location matters, are we appearing within the markets we serve?
Context: Why has the AI included us in the response?
Accuracy: Is the AI representing the business, product or service correctly?
Stage of the journey: Is the person learning about a subject, researching options, comparing providers or actively looking to buy?
This becomes particularly important because AI users aren’t necessarily searching in keywords.
A traditional search might be:
commercial boiler servicing Milton Keynes
An AI prompt could be:
We’re responsible for a large commercial building in Milton Keynes and need a company that can service the existing boiler and provide ongoing planned maintenance. Who should we speak to?
There is considerably more context and intent within the second query.
Simply knowing that our business appeared doesn’t tell us the whole story.
Understanding why it appeared tells us considerably more.
Not All AI Visibility Is Equal
In our AI Search Jargon Explained guide, we made an important distinction:
Mention ≠ Citation ≠ Recommendation
But we can take that further.
Not all mentions are equal.
Not all citations are equal.
Not all recommendations are equal.
A citation supporting a general definition isn’t necessarily equivalent to a citation supporting expertise in a specialist subject.
A passing mention of a brand isn’t necessarily equivalent to being included among relevant providers.
Being included in a comparison isn’t necessarily equivalent to being recommended.
And even a recommendation needs context.
After all:
“We wouldn’t recommend Company X for this particular requirement”
contains a company and a recommendation in the same sentence.
Probably not one to put on the monthly performance report.
The number alone can’t tell us the quality or context of the appearance.
Quality Over Quantity Doesn’t Mean Ignore Quantity
There is a danger here of swinging too far in the opposite direction.
If focusing purely on citation volume is potentially misleading, should we stop counting citations?
Of course not.
Quantity still matters.
If we’re being cited around highly relevant subjects and those citations increase from 20 to 100, that’s potentially useful information.
If we’re increasingly being recommended in response to commercially relevant prompts, we want to know.
If our AI Share of Voice is growing against genuine competitors across subjects that matter to the business, that could be valuable.
If more of our commercially important content is being used as a source, that’s worth investigating.
The lesson we learned from SEO wasn’t that traffic numbers were pointless.
It was that traffic numbers became considerably more useful once we understood the quality of the traffic behind them.
AI doesn’t kill the principles we’ve learned from SEO. If anything, it exposes why some of those principles mattered in the first place.
Quantity becomes meaningful once we understand what we’re counting and why we care about it.
And that distinction is increasingly being made within AI visibility measurement itself. Semrush explicitly separates visibility metrics such as mentions, citations and Share of Voice from the question of whether that visibility is contributing to revenue and ROI.
Source: Semrush: AI Visibility ROI
Beware the Shiny AI Dashboard
Marketing software has always been very good at giving us numbers.
AI visibility platforms are no exception.
1,426 citations.
38% Share of Voice.
AI Visibility Score: 74.
Excellent.
Now, what does 74 mean?
This isn’t an argument against AI visibility tools. Far from it. As this area develops, these platforms can give marketers access to information that would be extraordinarily difficult to collect manually.
But the availability of a metric doesn’t automatically make it a KPI.
Before getting too excited about a number, we need to understand what’s behind it.
Which AI platforms are included?
Which prompts are being tracked?
Why were those prompts selected?
Which topics are represented?
Which competitors are included?
Over what period is the comparison being made?
How is the score calculated?
And perhaps most importantly:
What does a change in this number tell us about the thing we’re actually trying to achieve?
If AI visibility rises from 60 to 75, that might be excellent news.
But before we celebrate, we need to understand what became more visible, to whom, for what and why that matters.
Otherwise, we’ve simply acquired a more sophisticated version of the traffic graph we were admiring 15 years ago.
Start With the Business Objective, Not the Metric
This is where we can turn the process around.
Instead of starting with:
Our software can measure AI citations, therefore AI citations are one of our KPIs.
Start with:
What is the business actually trying to achieve?
Imagine a commercial heating company wants to increase boiler servicing enquiries from businesses in Milton Keynes.
That gives us context.
We might then want to understand whether the company is appearing within AI-generated answers around commercial boiler servicing, planned maintenance, commercial heating contractors and related questions.
We’d be interested in whether those appearances relate to the right geography.
We’d want to know whether the business is simply being mentioned, whether its website is being cited or whether it’s being recommended as a potential provider.
And then the numbers become useful.
How frequently is that happening?
Is it increasing?
Which content is contributing?
How does visibility compare with genuine competitors?
A different business objective would create a different definition of valuable visibility.
For a research organisation, having its research repeatedly cited as an authoritative source could itself be extremely valuable.
For an ecommerce business, relevant product recommendations may carry greater importance.
For a local service business, being surfaced when someone is actively looking for a provider in its area could be considerably more commercially significant than hundreds of generic citations.
Quality is defined by the objective.
What Should You Measure Alongside AI Citations?
Once we’ve established that the visibility we’re measuring is relevant, we can start looking for other signals.
Are the right pages being cited?
Are we being associated with the right products, services or areas of expertise?
Are we appearing alongside businesses we genuinely consider competitors?
Are recommendations occurring around commercially relevant questions?
Is referral traffic from AI platforms reaching relevant pages?
Are people searching for the brand after encountering it elsewhere?
Are enquiries changing?
Are customers telling sales teams that they found or researched the business using ChatGPT, Google AI features or another AI tool?
We won’t always be able to join every dot.
Someone could discover a business within an AI answer, never click the citation, search for the brand two days later, visit the website, leave, come back directly the following week and then enquire.
Analytics may confidently award that conversion to another channel.
Marketing attribution wasn’t perfect before AI arrived.
It hasn’t suddenly become perfect now.
This isn’t simply theoretical. HubSpot gives a very similar example of an AI-influenced journey where someone encounters a brand through AI, later searches for it through Google and eventually converts through another channel. Last-click attribution can then give AI no credit at all.
Source: HubSpot: AI Search Visibility and ROI
We don’t need to prove that citation number 327 generated precisely £842.16 of revenue.
But we can build enough context around our AI visibility data to understand whether we’re becoming more visible in places that plausibly support what the business is trying to achieve.
A Simple Test for an AI Vanity Metric
Before celebrating an AI visibility number, ask:
1. What are we counting?
A citation? Mention? Recommendation? Prompt appearance? Visibility score?
2. Why does it matter?
What does this metric potentially tell us?
3. Is it relevant to our audience and business objective?
Are we appearing for something we actually want to be found for?
4. Do we understand the context?
Why are we appearing and how are we being represented?
5. What would an increase or decrease tell us?
Would we understand what had changed?
6. Would it influence a marketing decision?
Would the number help us decide what to investigate, improve, continue or change?
If we can’t answer those questions, the metric isn’t necessarily useless.
But we probably shouldn’t be presenting it as evidence of success.
And that distinction matters.
The Last Word
We’ve learned this lesson before.
More website traffic isn’t automatically better traffic.
More followers don’t automatically create more customers.
More impressions don’t automatically mean greater impact.
More rankings aren’t particularly useful if we’re ranking for the wrong things.
And more AI citations aren’t automatically better either.
But relevant traffic matters.
Relevant rankings matter.
Relevant visibility matters.
And relevant AI citations, mentions and recommendations absolutely can matter.
So this isn’t an argument to stop counting.
It’s an argument to understand what we’re counting, why we’re counting it and whether it has any relevance to what the business is actually trying to achieve.
AI visibility metrics can give us valuable information. As the tools, platforms and methodologies mature, our ability to understand that visibility will undoubtedly mature with them.
But we don’t need to forget everything we’ve already learned while we wait.
Are AI citations the new vanity metric?
They can be. Particularly if you’re focusing on quantity without relevance or context.
Count the citations. Just make sure they’re citations worth counting.
Part of the Marketing Clarity Series
This article is part of the Marketing Clarity series from The Last Hurdle, exploring the thinking behind clearer, more effective marketing.
From visibility and customer journeys to AI discovery and meaningful measurement, the series looks beyond the numbers to consider whether your marketing is reaching the right people, for the right reasons, and contributing to what your business is actually trying to achieve.
