AI Citation Evaluation Metrics: Audit Support, Sources, Share

A citation audit answers three questions: does the cited page prove the claim, whose sources keep getting cited, and is your domain among them? Keep them as three layers (claim, source, brand) and never blend them into one "citation score". Most dashboards count links; the right unit is the claim.

The research is why we check instead of counting. A human audit of four generative search engines found only 51.5% of sentences fully supported by their citations, and only 74.5% of citations supporting their sentence. In ALCE, even the best models lacked complete citation support 50% of the time on one benchmark. These are older systems, but the failure pattern is the one to audit for.

What should a citation audit measure?

Measure claims, sources and your own domain separately. The claim layer asks whether each statement is supported. The source layer asks who is cited and how concentrated, current and independent they are. The brand layer asks how often you appear.

The unit is the atomic claim, not the answer. FActScore splits a generation into atomic facts and reports the share supported by a reliable source. "AEOeye audits ChatGPT and costs $29" is two claims with two evidence needs.

"Supported" means entailed: a reader of the passage would infer the claim is most likely true. Relevance is weaker, and authority is a different question.

How do you record a citation so it can be checked later?

Save the observation before you interpret it. Per answer: exact prompt, engine surface, locale, timestamp, raw text. Per citation: marker, raw URL, resolved URL, fetch status, hashed snapshot. Labels and scores point back to this record and never overwrite it.

• URLs, three forms. Keep the raw URL as shown, the resolved URL after redirects, and a normalized comparison key. A 301 only says the resource has a new permanent URI, not that the destination makes the same claim, so mark it same, revised, substituted, unreachable or undetermined. Canonicals are signals of varying strength; store them separately.
• Normalize narrowly. Lowercase scheme and host, drop default ports, resolve dot segments, remove the fragment from the key. Drop only named tracking keys (utm_*, gclid, fbclid, msclkid). Keep lang, page and file; they can select different documents.
• Markers. Code the form (inline number, footnote, source card, bare link, hover, bibliography), its position, and the text it purports to cover; if the interface never says, code scope as ambiguous. OpenAI's API returns url_citation annotations with character offsets, which may differ from what a client renders.
• Access status. Record what a permitted reviewer could open: readable, login, paywall, script-rendered, region-blocked, rate-limited, soft error. A 403 means the server refuses the request, not that the page is absent. A 200 is not evidence of support. Never bypass access controls.
• Dates. Published, modified, effective, crawled and accessed are different questions. Google calls byline dates a best estimate. Never turn a crawl time into a publication date.
• IDs and uncertainty. Link study, run, prompt (with a parent for follow-ups), answer, claim, citation, snapshot, annotation and reviewer by stable IDs. Add an uncertainty note per row; keep both labels when reviewers disagree.

A SHA-256 hash proves retained bytes did not change, not that the page or label is right. Never regenerate a digest over a silently edited file.

We publish runnable fixtures for the mechanical parts. All are synthetic regression tests, not data about any engine:

• URL normalization test suite: 20 adversarial cases whose output must byte-match the expected file; malformed input is rejected, not guessed.
• Citation parser test corpus: 24 cases, 27 claims, eight categories. Detecting a marker is not support.
• Evidence bundle validator: checks paths, byte sizes, SHA-256 digests and citation-ID closure. Change one byte and it must fail.

How do you judge whether a citation supports the claim?

Split the answer into atomic claims, open the cited passage with its surrounding text, and label the relationship before judging the source. A famous page that does not state the number is a failed citation.

Use seven labels:

A vendor page saying a product "may reduce the risk of data loss with backups" does not support "prevents data loss". That is supports-with-qualifier.

Score source quality on a separate axis (high, medium, low, with reasons). A first-party page can fully support a price and be weak evidence for a performance comparison. With several citations on one claim, judge each alone, then say whether they support it jointly. Have a second person label a sample; use automated judges only to triage.

Which aggregate citation metrics matter?

Report a small set, each with its formula and denominator. Correctness and coverage describe claims, concentration and freshness describe sources, and citation share describes your brand.

The Justice Department's definition of HHI is squaring each share and summing. Worked example: five citations split two, two, one give shares 0.4, 0.4, 0.2, so HHI is 0.36 and effective domains are 2.78. Our concentration calculator runs this on a 30-row synthetic file (seven hosts, HHI 0.206667, 4.84 effective domains). It counts hosts, not registrable domains.

We set no "healthy" HHI threshold: HHI was built to measure market concentration, so market thresholds do not transfer, and a concentrated set can be right when one primary source is the best evidence.

Are the cited sources independent?

Count evidence streams, not domains. Six pages repeating one press release are one source. Trace the claim upstream, label each page primary, derived or independent, and report URLs, hosts, brands, owners and independent origins as separate counts.

To trace one claim:

1. Rewrite it with subject, number, unit, date and scope.
2. Search a distinctive phrase and build a timeline of who published first.
3. Follow references to a release, dataset, filing or test.
4. Record whether each page added evidence or only prose.
5. Mark unresolved origins as unresolved.

PROV-O models entities, activities and agents, which maps to page, republishing and publisher. Never infer ownership from templates or CDNs; "unknown" is a valid value. Label each source by origin and function (official page, paper, dataset, news, community post, catalog), and treat a snippet as an aggregator until you open the page behind it.

Distribution is skewed. Across 11,000 queries on five systems, Answer Bubbles found Wikipedia and longer sources over-represented, social media under-represented. Provenance signing does not fix truth: C2PA makes no judgment on whether signed assertions are good or bad, only whether they are intact.

Are the cited sources current?

Freshness depends on the claim, so "newest wins" is a bad rule. A price needs this week's page; a definition may suit an older primary source. Set a threshold per claim type up front.

Keep the dates apart. Schema.org's datePublished means first publication, while a Last-Modified header only reports when the server believes the representation changed. Keep conflicting dates; undated is not fresh.

FreshQA found that models of every size struggle with fast-changing knowledge. To time how fast a fix reaches answers, separate four milestones: crawl, retrieval, citation, correct answer. We suggest daily checks for 30 days and storing each event as an interval: first seen at a 15:00 check after a clean 09:00 check means "between 09:00 and 15:00", not 12:00.

What are the common citation failures?

Sixteen failure labels group into four families. Label one primary failure per claim: each family needs a different repair.

Report examples beside rates, and keep missing citations separate from failed ones. Version the labels: a lower failure rate after a rubric change may be a different rubric, not a better engine.

What should a brand actually track?

Track your domain's citation share against competitors on a fixed set of buyer questions, per engine. That is the number you can move.

1. Domain citation share and the same for three or four competitors.
2. The cited third-party hosts, with concentration, so you see which publishers carry your category.
3. Claim correctness for facts about you: price, features, availability.
4. Update lag after you change a page.

Crawler access limits what can be cited at all. In AEOeye's crawler access study (October 2026, 384 sites with a readable or absent robots.txt), 27 of 38 news publishers (71.1%) blocked at least one AI retrieval bot, against 1 of 193 SaaS sites and 4 of 90 e-commerce and consumer brands (method).

It is a convenience sample fetched once on one day, and a robots.txt block is a request, not enforcement. OpenAI documents that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. The publishers an engine could cite for you may be restricting it, so check your own access with the AI crawler directory.

Next: what AI citation tracking is, how ChatGPT chooses sources, our audit methodology template and recommendation measurement guide. AEOeye's free audit asks ChatGPT three unbranded buyer questions and shows the real answers; the $29 report adds four more engines and ten questions. It is not a daily tracker: for hundreds of prompts monitored daily, tools such as Profound, Peec AI or Otterly fit better.