AEO Experiments: Test a Specific Change Credibly

Write a hypothesis that can fail

A useful hypothesis identifies the change, the expected outcome and the context where the effect should appear. Add clearer content is too broad to test because neither the intervention nor success is defined. An illustrative hypothesis might propose that adding an explicit product-limit explanation to selected support pages reduces a specified recurring misstatement in a fixed answer panel. That can fail: the misstatement may persist, another error may appear or the selected pages may never be retrieved under the observed conditions.

Keep the hypothesis tied to a plausible mechanism without claiming knowledge of an undisclosed platform pipeline. Clearer evidence may help a retrieved passage support a correct answer, but the experiment still needs to observe whether that happens. Use passage clarity to define the editorial change precisely. If the intervention combines new facts, rewritten headings, a redesign and additional links, the result concerns that bundle. It cannot establish which component mattered unless the design separately varies those components in a way the evidence can support.

Choose the experimental unit carefully

The unit receiving the intervention might be a page, a group of related pages or an entire site. It is not automatically the same as the unit being measured. You may change a page and measure several prompt responses that could cite it. Those responses are related observations around the changed unit, not necessarily independent experimental units. If several pages share a template or strongly interconnected subject, changes can spill across the group. Account for that relationship before treating every answer as a separate independent trial.

NIST's design guidance ties design choice to the objective and factors. Applied to AEO, choose a unit that matches how the intervention is delivered and where interference is likely. A site-wide template change leaves no unchanged page within that template as a clean control. A narrow paragraph correction may permit a more bounded comparison. Describe the scope honestly rather than calling any before-and-after chart an experiment simply because a content change occurred between the two screenshots.

Build a comparison that addresses alternative explanations

A contemporaneous comparison group can help distinguish a site change from broader platform movement. Choose pages or topics that are relevantly similar before treatment, not only those that happen to support the desired conclusion afterwards. Random assignment can reduce selection problems when there are enough suitable units and the intervention can be allocated independently. NIST's randomised design explanation describes that assignment principle. It does not remove practical concerns such as shared templates, related topics or uneven collection coverage.

When randomisation is infeasible, state the weaker design clearly. A matched comparison or interrupted time series can still be informative, but its interpretation depends on assumptions about what would have happened without the change. Inspect pre-intervention patterns and record events that affect one group differently. A competitor launch, seasonal demand or an unrelated technical outage can alter outcomes. The sampling bias guide helps identify selection issues, while the experiment should explicitly explain which alternative explanations the comparison can address and which remain unresolved.

Define the outcome and window before release

Select a primary outcome that matches the hypothesis. If the intervention corrects a factual misunderstanding, measuring only citation count may miss whether the answer became more accurate. If it improves discoverability, an outcome focused solely on on-site conversions may be too distant and sparse to interpret. Keep supporting diagnostics separate from the primary result. Define the grading rubric, eligible observations and failure handling in advance so that the analysis does not switch to whichever measure looks most favourable after the data arrives.

Choose an evaluation window with the intervention's practical timing in mind. A published page may not immediately appear in the external systems being observed, and the publisher may not know exactly when those systems incorporate it. Record release and verification dates, then apply a predefined observation schedule. Do not repeatedly extend the window until a positive screenshot appears. Prompt tracking supplies the stable panel and collection record needed here. Retain missing runs and repeated observations so the result includes coverage and variability as well as the preferred headline number.

Protect the intervention from unrelated changes

Keep a release record of exactly which pages changed and what changed on them. Preserve a before version so that later reviewers can inspect the treatment. Coordinate with other teams where possible to avoid simultaneous alterations to navigation, product claims or analytics collection on the same units. If such changes are necessary, record them rather than pretending the original intervention remained isolated. A test can remain useful after a complication, but the claim it supports may need to become narrower or more tentative.

The content governance guide can establish ownership of this coordination. Do not freeze urgent corrections merely to preserve a clean experiment; factual accuracy and reader needs still matter. Instead, explain that the intervention changed or that the test was interrupted. An illustrative trial of a new comparison section may need to stop when a product capability is retired. Continuing to publish an inaccurate control page for the sake of measurement would undermine the library's purpose and produce evidence about a situation that no longer represents the real product.

Analyse the difference, not only the attractive screenshot

Compare the predefined outcome across the planned units and periods. Show the underlying counts, collection coverage and uncertainty appropriate to the design. If the treatment group improves while the comparison group improves similarly, the evidence for a treatment-specific effect is weaker than a simple before-and-after chart suggests. If only one topic improves, investigate whether that subgroup result was expected or discovered after reviewing many possible slices. Exploratory findings can inspire a follow-up, but they should not be presented as the original confirmed hypothesis.

Repeated answers may be clustered by prompt, page and observation time, so avoid treating every response as fully independent when estimating uncertainty. A small apparent difference can be unstable, particularly when the number of changed units is small. Seek suitable statistical help when the decision depends on a precise causal estimate. The original research guide covers transparent evidence design more broadly. An honest experiment can conclude that the available data is inconclusive, even when the revised content is clearer and worth retaining for readers.

Decide what to keep and what to test next

Interpret practical importance as well as direction. A tiny change in a sampled visibility rate may not justify a costly rollout, while correcting a repeated material error may be worthwhile even with limited traffic evidence. Consider whether the intervention has independent reader value and whether it creates maintenance costs or new risks of confusion. Keep the result tied to the tested scope. Evidence from one topic, platform and window does not establish that the same technique will work across every page or answer engine.

Record negative and mixed outcomes alongside positive ones. A reusable experiment note should contain the hypothesis, treatment, comparison, measurement definition, timing, result and remaining alternative explanations. Use AEO reporting to turn that evidence into a decision rather than a promotional claim. A good experiment reduces uncertainty about a specific change. It does not need to produce a winning tactic every time, and it should never replace a careful conclusion with a ranking guarantee that the design could not possibly establish.

Sources and further reading

• NIST experimental design selectionExperimental design depends on the objective and factors being investigated.
• NIST completely randomised designsRandomised designs randomly assign factor levels to experimental units.