AI Search Audit Methodology Template: 8 Decisions That Matter

An AI visibility audit is trustworthy when a stranger can rerun it and arrive at the same denominators. That depends on eight decisions made before you collect a single answer. Skip one and your percentage becomes an anecdote. This is our working method, not a standard.

What makes an AI visibility audit trustworthy?

An audit is trustworthy when its claim is small, its conditions are written down and its denominators are visible. A bare "30% visibility" is a rumor; "18 of 60 eligible answers" is a result.

We hold ourselves to this. Our AI crawler access study (October 2026) fetched 391 sites once each, 384 usable for robots.txt numbers. It ships the per-site CSV and a method summary, and states it is a convenience sample.

How do you build the prompt set?

Build it from real buyer language, group it by intent, and freeze it before you see any answer. A set of easy branded questions measures recognition, not discovery.

• Source: support tickets, sales calls, search queries. Declare any synthetic prompts.
• Strata: report branded and unbranded prompts as separate panels.
• No quiet swaps: never replace a prompt because it returned no mention. Absence is the observation.

We sort questions into six intent groups: discovery, comparison, validation, risk, implementation and post-purchase; each is coded for a different signal, such as inclusion, competitor set or caveats.

Then audit coverage. Declare minimum active prompts per stage, persona, topic, locale and engine; the auditor counts only active rows. Its synthetic fixture (20 rows, 18 active) is built to fail three minimums, for example pricing at 4 of 5.

For paraphrases, "equivalent" means the same intent, entity, constraints, locale and answer format; "best" to "cheapest" changes intent. Keep natural variants, changed-intent controls and adversarial edits in separate strata.

Which conditions must you control?

Hold fixed whatever you are not measuring, stratify what buyers genuinely vary, and record the rest. Google lists location, language, device, recent searches and account personalization as reasons results differ between people, so one logged-in laptop is one buyer's view.

NIST defines repeatability and reproducibility by whether conditions are the same or changed, so decide up front which are controls, factors or strata.

Three rules apply. A different answer is an observation, not proof of personalization; rerun the contrast before naming a cause. Locale tests need a repeated baseline, then one changed column per run. And a new chat clears visible thread context, not necessarily account settings, so code contamination instead of assuming a clean slate.

Order matters too. Fantastically Ordered Prompts found that example order can swing results from near state-of-the-art to near chance, and good orderings did not transfer between models. Test order with balanced permutations and a seeded shuffle.

How many prompts and repeats do you need?

Size the sample for the claim, not for a pleasant row count. With no prior estimate, simple random sampling needs 385 independent observations for a ±5-point margin at 95% confidence. The formula is n = z² × p(1 − p) / MOE², with p = 0.5 as the conservative default.

Comparing two rounds is more demanding. Our calculator uses the normal approximation for two equal groups (planning scenarios, not measurements):

Rows are not automatically independent. Twenty prompts repeated three times give 60 rows, not 60 observations. The design effect, 1 + (m − 1)ρ, shows how clustering shrinks effective sample size; use it as a sensitivity check. For small samples or rates near 0 or 1, use an exact or Wilson interval.

Repeats estimate volatility, not buyer intents. Most brand audits cannot reach 294 per group, so call them descriptive and never rank engines on them. In our own benchmark four of five engines have fewer than 10 mentions, so we publish no per-engine league table.

How do you code answers consistently?

Write the codes before you collect, label independently, and report agreement beside the raw labels. We use three codes: not_mentioned, mentioned and recommended. Settle edge cases in writing: comparisons, "best of" lists, caveats, aliases, a brand only in a source link.

Keep mention, recommendation, citation, rank and sentiment separate, and treat "unranked" as not applicable, never last place. A link near an answer does not prove the source supports a claim; split compound sentences into single claims.

For two reviewers, Cohen's kappa is (observed − expected agreement) / (1 − expected). Fleiss' kappa handles three or more raters; Krippendorff's alpha handles missing labels. Artstein and Poesio's survey explains how their chance assumptions differ, so pick one before you see data.

Our synthetic fixture has 24 rows with 18 agreements: raw agreement 0.75, chance-expected 0.333, kappa 0.625. Four of six disagreements sit on the mentioned-versus-recommended line, so write that rule first. Skip universal kappa cutoffs; read the disagreements.

Keep each original label, add the adjudicated label as a third field, and never compute agreement on consensus labels. Our benchmark's "mentioned" flag came from an analysis model, not human double-coding, so we treat its rates as indicative.

How do you handle exclusions and missing data?

Never turn an observation you failed to get into a negative result. A timeout is missing; a clean answer that omits your brand is a valid zero. Give every expected case one status and keep the raw evidence.

• Missing, not a finding: not run, provider unavailable, timeout, refusal, access blocked, empty answer, parse failure, unresolved.
• A valid zero: citation absent on an answer that was captured and parsed.
• Excluded by rule: not applicable, duplicate, or a documented rule.

With E expected cases, X exclusions, O valid observations and U missing, E = O + U + X. Report the rate (positives / O) and coverage (O / (E − X)): 40% of 10 observed cases is not 40% of 100 eligible cases with 90 missing. When missingness is material, add scenario bounds: positives / (O + U) as the floor, (positives + U) / (O + U) as the ceiling.

A retry keeps a case eligible; an exclusion removes it from the analysis set. Give attempts their own IDs, predeclare retry limits, and tag each exclusion PRE or POST; report results with and without any found after you saw outcomes. The PRISMA flow diagram is a good model: counts at each stage, with reasons for exclusions.

Automate the checks. The CSV validator checks seven columns, unique run IDs, strict booleans and blank measured fields on unavailable rows. The SQL kit reports two citation rates, over all available answers and over cited answers only, each with its denominator.

How do you version an audit so comparisons stay valid?

Treat any change to prompts, eligibility, codes, surface or metric formula as a new version, and say whether old and new numbers compare. Semantic Versioning supplies a useful vocabulary (patch, minor, major) and one rule worth stealing: released contents must not be modified, only superseded.

An engine or index update is an external event, not a method change: log the date and model label, flag drift, and do not bump your version. Zenodo advises citing the version DOI for reproducibility, not the concept DOI.

What should the published report include?

Publish enough that a stranger can rebuild your denominators and see what you did not claim. Open with a disclosure block: question; scope; prompts per stratum with a hash; runtime (engine, surface, model label, locale, account, timestamps); capture policy; codes and reviewers; n/N, exclusions, uncertainty; versions; limits.

Package the evidence: README, protocol, prompts, environment record, raw answers, source captures, codebook, adjudication log, code, checksums, licenses, changelog. Keep raw and derived files apart; a checksum detects change, not truth, and a live rerun is a new observation. Add a dataset card in the spirit of Datasheets for Datasets: motivation, composition, collection and recommended uses, with collection date separate from publication date.

Claim only what the design supports: "this answer displayed this URL at 14:05 UTC" is an observation; "engines prefer this brand" is not. Grade evidence claim by claim, from direct observation down to anecdote, not by publisher prestige. The GEO paper reports visibility boosts of up to 40% in generative engine responses on its own benchmark; that is a result for that setup, not a rate your brand will see.

Our recommendation benchmark shows the pattern: 127 audits, 892 answers, 157 brand mentions (17.6%), three internal rows excluded, a stated snapshot window and a warning that engine samples are unequal. Its write-up is on the measurement hub; citation evaluation metrics covers the citation side.

What is the minimum viable audit for a small team?

Run a descriptive audit you can defend: a small frozen panel, plain logging, honest wording. It shows whether you are absent, not whether a five-point change is real.

1. Write 12 to 20 unbranded buyer questions from real customer language.
2. Pick the engines your buyers use; test signed out, in one locale, at the default tier, and log the model label.
3. Run each prompt three times in fresh chats on one day, logging time, prompt, full answer and links.
4. Have two people code every answer, then compute raw agreement and kappa.
5. Report n/N with exclusions as "descriptive", and rerun the same panel after four weeks.

For a quick baseline, our free audit asks three unbranded buyer questions to ChatGPT live and shows the answers in full; the $29 report covers 10 questions across five engines. For daily monitoring across hundreds of prompts, dedicated trackers such as Profound, Peec AI or Otterly fit better. See also measuring AI visibility and the AEO audit checklist.