Multi-Platform AEO Testing: Compare Answers Fairly

Define the comparison before opening the tools

A useful test begins with a decision. You might need to know whether a product’s installation limits are represented accurately across services, or whether your documentation appears for implementation questions. Those are different studies. A broad question such as “which AI likes our brand?” has no stable unit of success. State what finding would change your work: correcting an inaccurate claim, improving a missing source page, or choosing a platform for closer investigation.

Choose the surfaces deliberately. A consumer chat interface, a dedicated search mode and an API tool can expose different controls and outputs even when their branding overlaps. Claude’s web search documentation describes an API search tool with citations and configurable behaviour. It is evidence about that tool, not proof that a consumer session uses identical settings. Record the actual interface used rather than reducing every observation to a provider’s name. When comparing ChatGPT Search visibility with Perplexity citations, use the platform guides to identify what the visible evidence can actually establish before assigning the same score to both.

Use a fixed question set built around real reader tasks. Include the constraints that make answers useful, such as region, organisation size, product edition or technical environment. The question research process helps distinguish those constraints from decorative wording. If you test “best booking software” on one platform and “booking software for a two-person UK clinic” on another, differences in recommendations may simply reflect different jobs rather than a difference in source preference.

Decide what must match and what must be recorded

Match exact question wording, language, approximate observation time and conversation position wherever possible. A first message in a fresh conversation is easier to compare than a question following ten personalised follow-ups. Record whether search was explicitly selected, automatically invoked, unavailable or not observable. Do not silently treat an answer with no visible sources as a searched answer that found no useful website. Those states describe different opportunities for your content to appear.

Some conditions cannot be standardised. Account settings, location inference, model routing and staged feature availability may remain uncertain. Use explicit fields for known values and unknown values. A blank cell invites later assumptions; “not exposed by interface” preserves the actual limitation. NIST’s AI Risk Management Framework treats evaluation within the broader use of an AI system. Applying that principle here means describing the service as tested, including the context that could change its behaviour. The Gemini web answers guide and Microsoft Copilot guide help scope those platform records; they do not remove the need to document the exact experience tested.

Set an observation window that limits drift while remaining practical. For an illustrative study, three reviewers could divide platforms during the same working day using the same question list. They should follow one protocol for opening conversations and saving outputs. If one platform is unavailable until the following week, label that run separately. A tidy spreadsheet is not a reason to pretend the answers were produced under the same conditions when a product update or news event intervened.

Save evidence before turning answers into scores

Capture the exact question, complete answer, displayed citations, destination URLs, timestamp and any visible mode label. A screenshot can preserve interface context, but text is easier to analyse and verify. Save both when the interface makes citation mapping ambiguous. Keep source order if you plan to analyse prominence. A list of domains alone loses whether the link supported the main recommendation, a minor aside or a statement unrelated to the brand being measured.

Agree classification rules before reviewers see the aggregate results. Distinguish a brand mention from a recommendation, an owned-domain link from an independent source, and a correct product fact from an unsupported claim. Give reviewers a borderline example. An answer can mention your company negatively, cite your homepage without supporting its claim, or recommend your category without naming you. Combining those states into one “visibility” score makes improvement difficult to interpret and can reward the wrong outcome.

Separate collection failures from observed negatives. If an interface times out, the test has no answer to score. If a valid answer contains no brand mention, that is an observed absence. If search runs but citations cannot be opened, source verification is incomplete. Preserve those statuses rather than filling them with zero. The citation quality guide explains why an existing link still needs checking against the nearby claim before it becomes evidence of useful representation.

Create a short adjudication record for disputed classifications. If one reviewer calls an answer a recommendation and another calls it a neutral mention, preserve the sentence and explain which rule resolved the disagreement. Reuse that decision for equivalent cases later. This makes the dataset more consistent without hiding judgement behind an apparently objective number, and it gives future reviewers a concrete precedent when an interface presents a recommendation in an unfamiliar format.

Use denominators that show the opportunity

Suppose an illustrative test contains twenty questions. Platform A produces twenty valid answers and searches for twelve. Platform B produces eighteen valid answers and searches for all eighteen. Four owned-domain citations on each platform do not establish equal performance. Report four out of twenty valid answers for A and four out of eighteen for B, then separately report citation occurrence among searched answers if that is meaningful. Readers need both the count and the rule behind the denominator.

Repeated runs help describe variation, but repeated answers to one question are not twenty different reader needs. Retain both question-level coverage and run-level occurrence. Otherwise a heavily repeated branded query can dominate the average while an important unbranded problem remains unanswered. Sampling bias becomes especially relevant when the question list was selected because the business already performs well on it. A fair comparison needs the inconvenient questions that matter to customers as well.

Show platform results before any combined total. A weighted total requires a defensible reason for the weights, such as the audience you are actually studying. Equal weighting is also a choice, not neutrality by default. If you lack reliable usage evidence, label the overall figure as an equal-weight test index. Do not call it market share. The distinction helps decision-makers avoid allocating effort as though a small controlled sample measured the behaviour of every potential customer.

Turn differences into focused follow-up work

Investigate a pattern through examples. If one platform repeatedly describes an old plan limit, inspect the cited pages and date-sensitive passages. If another never searches for that topic, rewriting a citation-friendly opening may not address the immediate mechanism. If answers disagree but cite the same page, the source wording may be ambiguous. Use passage clarity to examine whether conditions and numbers remain connected when a paragraph is read on its own.

Publish a short methods note with the resulting comparison. Include the question-set version, interfaces, collection dates, valid-run counts and classification definitions. Keep the raw observations accessible to the people making decisions. Then choose a narrow next test rather than rewriting the entire site because one platform produced a disappointing answer. The purpose of cross-platform testing is to locate specific differences worth explaining. A report earns trust when a second reviewer can follow the evidence and reach a compatible interpretation.

Sources and further reading

• Claude web search toolThe API web search tool supplies citations and supports search configuration.
• NIST AI Risk Management FrameworkThe framework addresses trustworthy AI design, use and evaluation.