AI Brand Recommendation Annotation Codebook: Labels and Examples

An AI brand recommendation study is only as reproducible as its labels. This codebook records whether an answer recommends, qualifies, mentions, criticizes, excludes, or omits a brand, while keeping evidence visible.

These labels are AEOeye's operational proposal, not a universal industry standard or a product's internal scoring system. Version the rules, preserve the raw answer, and disclose adaptations. The AEOeye audit methodology template provides wider protocol context.

Table of contents

• What is the unit of annotation?
• Which primary labels should reviewers apply?
• How should rank and sentiment be recorded?
• How should citations and evidence be coded?
• How do difficult answer structures change the label?
• How should teams adjudicate disagreements?
• What can this codebook not prove?
• What are the common questions?

What is the unit of annotation?

The atomic unit is one target brand in one captured answer to one prompt, at one recorded runtime. Store the prompt, answer, engine or interface, timestamp, locale, model label when exposed, and response identifier before coding.

Do not use a screenshot alone as the dataset. Preserve response text and visible links, then mark the exact brand span and the sentence or list item that gives it meaning. This follows the evidence-first logic in citation-focused work such as ALCE.

One answer can therefore produce several rows:

The row describes what was observed, not why the engine produced it. Do not infer hidden ranking factors.

Second image: Pexels photo by fauxels, Pexels profile.

Which primary labels should reviewers apply?

Apply exactly one primary label to each target brand per answer. The label describes the answer's stance toward that brand, not whether the underlying product is objectively good.

Use competitor_only only when the target was in the study's defined set but did not appear. It is not a negative statement about the missing brand. A nonexistent brand name belongs in an entity-status field, not a recommendation.

How should rank and sentiment be recorded?

Rank is the position the answer visibly assigns; sentiment is the tone toward the target. Keep both separate from the primary label because a qualified first-place recommendation and a neutral third-place mention are materially different observations.

Record an integer rank only when the answer supplies an ordered list or explicit ordering such as “my first choice.” Use unranked for prose, and not_applicable for refusals or absent brands. Ties receive the same rank and a tie=true flag; do not break a tie by guessing order.

Use four sentiment values: positive, neutral, negative, and unclear. Do not convert “best for X, weak for Y” into a universal verdict; record the qualifier in notes.

For example, “AtlasMemo is second, but its export controls are limited” is qualified_recommendation, rank 2, and negative only if sentiment is defined as overall evaluative tone. Otherwise use mixed as a declared extension. Never add a fifth value mid-study.

How should citations and evidence be coded?

Evidence status answers whether the observed recommendation is supported by an accessible source, not whether the recommendation is true. This distinction matters because FActScore evaluates fine-grained factual claims, while an annotation study first needs to preserve what the answer actually said.

Use these fields:

• citation_present: yes, no, or unclear.
• citation_attributed: yes when a link is connected to the target claim, otherwise no.
• evidence_status: supported, partialsupport, contradicted, inaccessible, unsupported, or notapplicable.
• evidence_url: the raw visible URL, plus a resolved URL if redirects are followed.
• evidence_note: the claim span and the reason for the status.

“Northstar Notes supports SSO [link]” is supported only if the linked page actually establishes that capability. A relevant pricing page does not entail a security claim. If the link returns an error, mark inaccessible, not unsupported; if no usable source exists, mark unsupported. Keep the recommendation label unchanged.

This prevents citation laundering: a nearby link is not automatically evidence for every sentence. NIST AI measurement guidance supports documenting design and limits, not treating proximity as proof.

How do difficult answer structures change the label?

Review the complete answer and nearby turns. Tables, bullets, negation, conditionals, follow-ups, duplicate mentions, and hallucinated entities can change a short brand span's meaning.

For multi-brand tables, code each row independently and preserve column headers. “Best for price” is a qualified recommendation, not a universal winner. For unordered prose lists, use unranked.

Negation controls the label: “Northstar is not a bad option” is not an explicit recommendation. If the answer says “I would not choose Northstar unless offline mode is essential,” code a qualified recommendation only when the conditional choice is clear; otherwise use negative_mention with the exact span in notes.

Follow-up turns share a session but retain separate answer IDs. If a later answer corrects an earlier one, annotate both and link them with supersedes_answer_id. Duplicate mentions count once for presence; distinct claim spans can have separate evidence notes.

If the model names a brand that the study cannot verify, retain the observed text and set entity_verified=false. Do not replace it with “no answer.” Hallucinated or ambiguous entities are findings about the observation and should be escalated for adjudication.

How should teams adjudicate disagreements?

Train reviewers on a small shared set, then have at least two reviewers independently code a defined sample. Disagreements should be resolved against the written rule, with an adjudicator recording the final label and the reason for changing either reviewer's decision.

Keep a versioned decision log for edge cases: conditional tables, aliases, ties, and inaccessible citations. Report double-coded items, disagreements by field, and the adjudication rule. Agreement is a process diagnostic, not proof of validity.

The NIST AI Risk Management Framework is governance context, not a label prescription. AEOeye's visibility score methodology is likewise an implementation example.

What can this codebook not prove?

This codebook describes observed answer behavior for a declared prompt panel and runtime. It cannot prove market share, sales impact, causal lift from a content change, every user's experience, or the factual truth of every recommendation. It also cannot make two engines directly comparable when their interfaces expose different retrieval, personalization, or citation behavior.

Treat the output as an operational dataset: useful for consistent review, trend tracking, and follow-up investigation. Publish the prompt scope, exclusions, version, denominators, and missingness. If you change the labels or sampling frame, start a new codebook version rather than silently breaking the time series.

What are the common questions?

Should every recommendation have a citation?

No. Citation presence and recommendation stance are separate fields. An uncited recommendation can still be coded as observed, with evidence marked unsupported or not applicable.

Is a first-listed brand always ranked first?

No. Code rank only when the wording or structure establishes order. A prose paragraph or unordered table can mention a brand without assigning rank.

How should a tie be scored?

Assign the same rank to tied brands and record the tie. Never invent a deterministic order from visual placement alone.

Can AEOeye's labels be used in another study?

Yes, as a named operational starting point. Copy the version, preserve the definitions, disclose adaptations, and do not describe the codebook as an ISO, NIST, or engine-standard taxonomy.

For a baseline of how your brand appears in buyer questions, run a free AEOeye audit. Keep its prompt, date, engine context, and answer evidence alongside any researcher-coded dataset.