Original Research for AEO: Publish Evidence Readers Can Inspect

Choose a question that evidence can answer

A research project should begin with a question narrower than the claim you hope to promote. “Does clear documentation improve business performance?” contains too many undefined terms for a small study. “Which permission conditions are missing from a defined set of public export guides?” is an observable question. It can produce useful evidence without pretending to explain every downstream outcome. Narrow scope makes collection rules clearer and gives readers a better basis for deciding where the findings apply.

Identify the unit of observation. It might be a webpage, organisation, support request, answer or individual respondent. These units are not interchangeable. Ten answers generated from one prompt are ten outputs but only one question context. Five pages from the same company may share one template. Define the unit before counting, because the apparent size of a dataset can exaggerate its independence. The sampling bias guide explains how selection and repetition can distort the story a dataset seems to tell.

Choose a design suited to the question. A structured content audit can describe published information. A survey can report what respondents say. A usability observation can reveal where participants struggle with a task. None automatically proves that one factor caused another. If the question concerns whether a specific content intervention caused a change, use the separate AEO experiments guide. Original research is broader than causal testing, and a descriptive study can be valuable when its claims stay descriptive.

Specify the sample before collecting attractive examples

Describe the population you care about and the source from which you will select observations. A list of your customers is accessible, but it cannot automatically represent all businesses. Public pages discovered through one query reflect that discovery route. Explain inclusion and exclusion criteria in concrete terms, such as language, product category and date range. If access limitations narrow the sample, record that constraint rather than quietly treating the available subset as the original intended population.

Decide how to handle missing or inaccessible observations. If a page fails to load, that is not necessarily evidence that the information is absent from the organisation’s documentation. If a respondent skips a question, the denominator for that result may differ from the overall response count. Preserve missingness as its own state. This prevents a convenient coding shortcut from turning unknown cases into negative findings and changing the meaning of percentages later in the report.

A small study can still be useful if it is presented honestly. An illustrative audit of forty selected guides may identify recurring types of unclear permission wording. It cannot establish the prevalence of that problem across every industry unless the sampling design supports that inference. Resist the urge to compensate for a limited sample with a universal headline. The strongest contribution may be a carefully documented taxonomy of failure cases that helps other publishers improve their own explanations.

Make measurement rules reproducible

Write a coding guide before the main collection. If you assess whether a page states a permission requirement, define what counts: a named role, a linked prerequisite, a vague mention of access, or some combination. Include borderline examples and explain how they should be classified. A rule that exists only in the researcher’s head is difficult to apply consistently. It also becomes easy to change unconsciously when a surprising early result begins to shape expectations.

Run a small pilot to expose unclear categories and collection problems. Revise the method if necessary, then record what changed before the main study. If more than one person codes the material, compare their decisions on the same examples and resolve disagreements through the rule definitions. Do not treat agreement as proof that the underlying construct is perfect. Reviewers can consistently measure the wrong thing. The point is to make judgement explicit enough that someone else can understand and challenge it.

AAPOR’s Transparency Initiative promotes disclosure of research methods and distinguishes transparency from a judgement of methodological quality. That distinction is useful beyond surveys. Publishing a spreadsheet makes a study inspectable, but it does not repair a biased sample or an unsuitable measurement. Treat transparency as necessary evidence about how the work was done, while explaining the separate reasons the method is appropriate for the question you are asking.

Report the result with its denominator and limits

Show counts beside percentages. In an illustrative audit, “twelve of forty reviewed guides named the required role” lets readers understand the observation directly. If five guides were inaccessible and excluded, report that fact and explain whether forty refers to the original sample or the usable subset. Do not choose the denominator after seeing which version produces a more dramatic figure. The denominator is part of the meaning of the result, not a cosmetic formatting choice.

Keep associations, examples and causal explanations separate. If pages with named authors also contain clearer permission guidance, the observation does not prove that author bylines caused clarity. Both may reflect a stronger editorial process or a different type of publisher. Offer explanations as hypotheses when appropriate, and make clear what additional research would distinguish them. Readers can benefit from a plausible interpretation without being asked to accept it as a demonstrated mechanism.

Discuss uncertainty in a form appropriate to the design. A formal margin of error has assumptions and should not be attached casually to a convenience sample. For a small qualitative study, describe the range of observed cases and the limits of transfer instead of manufacturing statistical precision. Source corroboration can help compare your findings with genuinely independent evidence, but agreement with another study does not remove differences in population, period or method.

Publish an evidence package, not only a headline

The article should present the question, method, main findings, representative examples and limitations in a readable sequence. Put the core finding near the beginning, then provide enough detail for scrutiny. Google’s people-first guidance encourages original information and analysis. Your contribution should be the new evidence and its explanation, rather than a dramatic claim repeated across several sections with the actual method hidden in a footnote.

Provide supporting material where it is appropriate to share: a data dictionary, collection dates, coding rules and aggregated or redacted observations. Do not publish personal or confidential information merely to make the study appear transparent. Explain any restrictions and provide the most useful safe level of detail. A stable methodology section also helps future readers distinguish a later correction from a change in the research design. Editorial trust depends on making those differences visible rather than silently rewriting the record.

When the study is repeated, preserve comparability deliberately. A changed sample or coding definition can explain a different result even if the headline metric has the same name. Publish the change and avoid presenting the series as a continuous trend without qualification. Use content freshness to keep the article’s dates and historical scope accurate. Useful original research gives readers a reliable account of what was observed, how it was observed and which decisions the evidence can reasonably inform.

Sources and further reading

• AAPOR Transparency InitiativeMethod disclosure enables scrutiny but does not itself certify research quality.
• Google people-first content guidanceThe guidance asks publishers to provide useful original content for people.