EWR Digital
A laptop showing the words SEO Agency on its screen, between a large magnifying glass and a brass scale of justice on a white desk.
SEO & LLM Visibility

Two Tools Measured One Brand's ChatGPT Visibility. They Were 25 Points Apart.

By Matthew BertramNovember 17, 2026

In October, two tools measured how often ChatGPT names one of the brands our tracker follows when someone asks it to recommend a marketing agency. One put the figure at 34%. The other put it at 9%. We built the second tool ourselves and have paid for the first since January, so if we lean toward either number it is the one from our own instrument, and you should read what follows with that in mind.

Data as of October 2026: ChatGPT and Perplexity answers collected by both tools in September and October 2026, compared question by question.

The natural reading is that one tool is broken, and the natural suspect is whichever one gives the number you like less. Rachel Hernandez, writing for the SEO company The HOTH in September, described the general pattern plainly: "Run your brand through two AI visibility platforms in the same week and you will very likely get two different scores." Different is one thing. Twenty-five points on the same engine in the same month is another.

Perplexity is where the broken tool theory runs into trouble. In the same month, on Perplexity, the two tools put the brand at 42% and 40%.

So we laid the two question lists side by side.

Forty questions on one list, thirty-two on the other

The other tool asks 40 questions, written the way a buyer might type them, many naming an industry and a city. Ours asks 32, some of the same kind and some broader, such as "best B2B SEO agency." Not one question appears on both lists. Neither list contains the brand's name either, so the inflation Hernandez warns about from branded questions, which she says "can move the headline score by 10 points or more on their own" in a 30-question set, is not what is going on here.

Both numbers mostly measure one industry.

ChatGPT naming the brandOther tool, Sep and OctOur tracker, Oct
Questions on the list4032
Questions that name the brand's core industry156
Named the brand on those questions72 of 105 answers (69%)26 of 72 answers (36%)
Named the brand on every other question21 of 175 answers (12%)8 of 312 answers (3%)
Overall rate33%9%

We combined the other tool's September and October because it ran only twice in October, on the 7th and the 13th, and October alone came to 27 of 80 answers, the 34% above. Six of its industry questions named the brand in all seven answers they got across the two months, while 22 of its 40 questions never named it once, so most of its headline rests on a short stretch of the list. Ours looks the same. Three industry questions produced 26 of the brand's 34 mentions.

What the mix explains, and what it leaves

Since the industry questions carry both numbers, the share of each list given to them moves each headline. Fifteen of 40 is 38%. Six of 32 is 19%. If our tracker gave the industry questions the weight the other tool does, and kept its own rates within each group, its 9% would come out at about 15%. Run the other way, the other tool's 33% would drop to about 23% on our mix of questions.

That closes somewhere between a quarter and nearly half of the gap, depending on which direction you reweight.

The rest sits inside each group. On the industry questions the other tool found the brand named 69% of the time and ours found 36%; on everything else, 12% against 3%, which means ChatGPT named the same company two to four times as often on one tool's questions as on the other's, in the same weeks.

We can name the differences but not weigh them. Our tracker queries ChatGPT through a data provider using gpt-5-mini, and the other tool does not say which model it uses. We ask each question 12 times on a single day; it asks each question once per weekly run. About a fifth of its ChatGPT answers, 63 of 280, were run as if the user were in Houston, where ours are set to the United States as a whole.

Two explanations we can rule out. Every one of its 280 ChatGPT answers came back with sources, so both tools had web search switched on. And both measured inside the same window in which the server hosting the brand's old site was blocking OpenAI's crawlers, a problem we describe in how to measure an AI citation rate, so the block cannot explain a gap between them. Which of the remaining differences carries the other half of the gap is something we cannot see from outside the other tool.

Two ways to ask about AEO for law firms

The sharpest single comparison is on the law firm questions. The other tool asks ChatGPT "Best AEO agencies for law firms?" and got the brand's name in 5 of 7 answers across September and October. Our tracker asks "AEO agency for law firms" and got it in none of 12 answers in September and none of 12 in October. On the other tool's own list, "Which agencies get law firms cited in ChatGPT and AI search answers?" named it in 0 of 7.

To a person, those questions mean nearly the same thing. The engine disagreed.

Our reading, which two questions cannot prove, is that a request for the best agencies, plural, invites a list, and a list has room for a brand that has published its own ranking of SEO and AEO companies for law firms, while a request for an agency invites a short answer that goes to the names the model is surest of. We saw something similar on Perplexity, where the same question drifted from lists of agencies toward agencies' own pages over six weeks, which we traced in how long a Perplexity citation lasts. Hernandez's summary fits both: "Your score is a property of your prompt list at least as much as it is a property of your visibility."

Questions for whoever sends your firm's report

If your firm's SEO report now carries an AI visibility number, the number is only as good as the list behind it. Ask for the list. Ask who wrote it and whether it was written before or after anyone looked at the answers, because, in Hernandez's words, "it is easy to write a list that flatters you without meaning to."

Then ask whose questions they are. A list written by marketers asks about marketing, while your clients ask about their own situation: who handles truck accident claims in Harris County, whether a lawyer charges anything up front, which firm a neighbor used after a crash on the Beltway. A score built on the first kind says little about the second.

Ask for the per-question table, not just the headline. If a handful of questions supply three quarters of the mentions, as the industry questions did on both lists, the headline is really a report on that handful, and the firm should know which handful it is. When we run an AI visibility audit for a firm, that table comes first and the headline comes last.

Last, ask whether the list has changed. Ours grew from 10 questions in mid-August to 32 by September, and we do not compare numbers across that change. A report that does is partly measuring its own edits.

The side-by-side we have not run

The clean test is to put both lists through one tool on the same day, with the model, location and schedule held fixed, so that the list is the only thing that differs. Whatever gap survives that test belongs to the instrument, and whatever disappears belonged to the questions, which would finally tell us how much of any AI visibility number describes the firm and how much describes the people who chose what to ask.

We have not run it. If your firm or agency tracks AI visibility against a question list you would trade for ours, we would like to put the two through the same instrument and publish what changes.

Matthew Bertram
About the author

Matthew (Matt) Bertram helps energy and industrial companies get found, trusted, and chosen as AI reshapes how buyers decide. As owner and CEO of EWR Digital, he leads the agency's AI search and SEO work for oil and gas and other capital-intensive industries; he is also President of ModalPoint. He is also Chief Marketing Officer of the Oil & Gas Global Network (OGGN) and the author of multiple books, including LLM Visibility: A Decision-Grade System for Winning AI-Mediated Discovery and the co-authored Oil & Gas Sales & Marketing: The Energy Growth Playbook for Oil and Gas Leaders. He is a member of the American Petroleum Institute's Houston Chapter and the International Association of Privacy Professionals (IAPP).

Next Step

Want to know how your site actually shows up in search and AI answers?

Free Discovery Call