AI Answer Snapshot Diff Tool: Track Mention, Recommendation, and Citation Changes

The reliable way to explain a changed AI-search result is to diff two dated, coded exports—not to compare screenshots by eye. This page ships a runnable Python tool and populated fixtures that join on prompt_id + engine, then report availability, brand mention, recommendation label, rank, citation-domain additions/removals, and prompt-set gaps. The before and after data are completely synthetic; they are not live engine results or a benchmark.

Table of contents

• What does a snapshot diff prove?
• Why is the stable join key important?
• Which answer fields should be compared?
• What is in the runnable package?
• How do you run the exact byte-match check?
• How should teams use diffs before aggregates?
• What are the limitations and sources?
• FAQ

Image attribution: ThisIsEngineering on Pexels. This editorial image is illustrative, not a test screenshot.

What does a snapshot diff prove?

A diff proves only that two supplied records have different coded values under a stated comparison contract. It gives a small, reviewable change log: “rank moved from 1 to 2” or “this citation domain disappeared.” It does not establish causality, factual correctness, provider quality, or a population trend.

That modest claim is a strength. A screenshot comparison encourages memory, and a single aggregate can hide whether the denominator changed. A row-level output keeps the observation and its interpretation separate. Record the snapshot date and timezone in RFC 3339 form, plus the visible engine/model, locale, account state, prompt version, and extraction rubric. RFC 3339 is a syntax for timestamps, not evidence that two captures were collected under equivalent conditions (RFC 3339).

The package is a documentation-based, reproducible example. It has not queried ChatGPT, Perplexity, Gemini, Google, or any other live answer surface. The records use example domains and synthetic labels so the expected output can be inspected without exposing a real brand or pretending to have run a provider test.

Why is the stable join key important?

Join the same intended question on prompt_id and engine; never join by row position or answer text. The pair preserves the study design: prompt_id identifies the prompt version, while engine keeps separate answer surfaces from being blended.

For example, a recommendation changing on synthetic-engine-a is not the same observation as a new row on synthetic-engine-b. A prompt can also be edited while its human-readable wording remains similar. Give prompt definitions immutable IDs, and version them when wording, locale, or instruction context changes.

The script takes the union of keys in both snapshots. If a key appears only before or only after, it emits prompt_set_gap instead of inventing a false negative. This follows the provenance principle that an analyst should be able to trace a derived statement back to the records and activity that produced it (W3C PROV-O).

Which answer fields should be compared?

Compare fields that map directly to a measurement decision, and preserve their raw coded values. The fixture uses four scalar fields and one set-like field:

Field	What the diff says	What it does not say
availability	Whether a coded answer was available	Why it was unavailable
brand_mention	Whether the target brand was marked present	Whether the mention was favorable
recommendation_label	A controlled label changed	That the label is objectively correct
rank	A numeric position changed, appeared, or disappeared	That rank is comparable across engines
citation_domains	Domains were added or removed from the coded set	That domains are independent owners or supporting evidence

Keep false, null, and an empty list distinct from a missing record. An available answer with brand_mention=false is a measured non-mention. A null rank can mean “not ranked” under your rubric. A prompt-set gap means the observation is absent and needs a sampling explanation. NIST describes AI measurement as context-dependent evaluation, so those context and denominator decisions belong in the report, not in a hidden cleaning step (NIST AI Measurement and Evaluation).

Image attribution: Katerina Holmes on Pexels. This editorial image is illustrative, not a live-answer capture.

What is in the runnable package?

Open the complete AI answer snapshot diff README. It documents:

• diff_answers.py, a dependency-free Python 3 comparator.
• before.json and after.json, populated synthetic snapshots.
• expected-diff.csv, the byte-level output contract.
• README.md, scope, command, and interpretation rules.

The implementation uses Python’s standard json and csv modules. JSON is convenient for preserving arrays such as citation domains; CSV is deliberately used for the output because it is easy to review, diff, and import into a spreadsheet or database (Python json, Python csv).

How do you run the exact byte-match check?

From the resource directory, run:

python3 diff_answers.py before.json after.json > generated-diff.csv
cmp --silent generated-diff.csv expected-diff.csv

The second command exits 0 only when header, row order, commas, empty cells, and line endings all match. The expected fixture includes a rank change, recommendation and availability changes, citation additions/removals, and two prompt-set gaps. That makes the contract more useful than a “script ran” claim: reviewers can inspect each expected row and see which rule produced it.

For a real audit, save the raw exports and hashes, then attach the diff to a provenance record. Do not overwrite the earlier snapshot. The diff is a derived artifact; the raw answer, extraction decision, collection timestamp, and settings remain the evidence a reviewer needs to challenge a row.

How should teams use diffs before aggregates?

Read the diff before calculating rates. First check whether the prompt-engine denominator is stable. Then classify changes: answer availability, brand visibility, recommendation, rank, and sources. Only after that should you aggregate by engine, prompt intent, date window, or brand.

Consider a simple example from the synthetic output. One row moves from rank 1 to rank 2 while retaining a recommendation; that is a presentation change, not a recommendation flip. Another row becomes unavailable and loses its citation domain; treating that as “no mention” would mix collection failure with a measured answer. A third prompt is absent from the newer file; the correct action is to investigate the prompt set, not silently add a zero.

For production use, add an explicit schema version, collected_at, model or product label, locale, and evidence URI to each record. Keep recommendation labels controlled and document whether a citation domain is derived from a URL normalization policy. Domain additions can reflect URL extraction or normalization changes rather than new sources. A diff reveals where to investigate; it does not remove the need for adjudication.

What are the limitations and sources?

This tool cannot retrieve answers, prove that an engine changed, infer why a citation was selected, validate citation support, identify source ownership, or determine whether a recommendation is good. It also cannot repair a weak sampling frame, an inconsistent brand-alias policy, or a changed prompt. If the extraction rubric changes between snapshots, label that as a measurement change and consider rerunning the earlier export under the new codebook.

The primary references are RFC 3339 for timestamp syntax, W3C PROV-O for provenance concepts, NIST AI Measurement and Evaluation for context-dependent measurement, and the Python json and csv documentation for the standard-library implementation. The fixtures and expected rows are AEOeye synthetic test data; they are not observations, benchmark results, or claims about any provider.

FAQ

What does an AI answer snapshot diff prove?

It proves that two supplied, coded snapshots differ at specified prompt-and-engine rows and fields. It does not prove causality, correctness, provider quality, or a durable trend.

Why join on prompt_id and engine?

That stable pair identifies the same intended prompt on the same answer surface. Joining on row order or answer wording can compare different questions and engines without showing the mistake.

Should a missing record count as zero visibility?

No. Emit a prompt-set gap and investigate collection or sampling. A missing row is not equivalent to an available answer that explicitly records no mention or no citations.

Are the included snapshots live engine results?

No. Before and after are completely synthetic fixtures. They test deterministic comparison mechanics only; replace them with documented exports before making claims about a real engine, brand, or date.

If you need a prompt-level baseline before maintaining snapshots, run a free AEOeye audit, retain its raw evidence under your study policy, and then compare future exports with an explicit join key.