ailathato.hu / Articles / Guide

How to Verify Claims AI Makes About a Company

A step-by-step guide to auditing AI-generated statements using claim extraction, human coding, and source verification protocols.

By Zoltán Varga2026-11-056 min read

Breaking Answers into Individual Claims

Start by treating an AI response as a collection of distinct statements rather than a single block of text. A single answer often contains multiple assertions about a company's products, history, or partnerships. You must isolate each brand-linked claim into its own unit before attempting verification. For example, separate the statement 'Company X offers product Y' from 'Product Y was launched in 2023'.

Research protocols define a claim unit as a specific assertion linking a brand to a fact, capability, or event. This granularity allows you to identify which parts of an answer are correct while others fail. An AI might correctly identify a company's headquarters but fabricate a specific service line. Without breaking the text apart, you cannot measure the true error rate or understand the nature of the mistake.

Use a structured approach to list every claim found in the output. Record the exact wording used by the AI for each point. This creates a clear dataset for the next steps of your audit. Do not skip vague statements; label them as ambiguous if they lack specific details that can be checked against external records.

Focus on claims that carry the highest risk for your organization. Statements about regulatory compliance, financial performance, or product capabilities require immediate scrutiny. The goal is to create a clean inventory of assertions that can be tested one by one. This method prevents the confusion that arises when trying to validate a long, flowing narrative as a single unit.

• Identify every sentence containing a brand name and a factual assertion.
• Separate compound sentences into distinct claim units.
• Record the exact phrasing used by the AI for each claim.
• Flag vague or ambiguous statements for further review.

Labeling Verified, False, and Unverifiable Claims

Assign a specific label to each extracted claim based on your investigation. The standard categories include accurate supported, accurate unsupported, and various types of hallucinations. Accurate supported means the claim is true and backed by a source provided in the output. Accurate unsupported indicates the claim is likely true but lacks a citation in the AI response.

False attribution occurs when the AI assigns a fact to a brand that does not belong to it. Fabricated capability describes a nonexistent product or service invented by the model. Fabricated source refers to a citation that looks real but points to a non-existent document or URL. Misattributed source happens when a real document exists but does not support the specific claim made.

Confident false claims present a unique challenge. These are statements written with strong certainty but lacking any evidence. They often appear in answers about complex topics like regulatory compliance or ethical standards. You must mark these as unsupported confident claims if no verifiable evidence can be found. This distinction helps differentiate between honest uncertainty and deceptive fabrication.

Maintain a neutral stance during the labeling process. Do not assume the AI is lying; treat every claim as a hypothesis to be tested. If a claim cannot be confirmed or denied with available information, label it as ambiguous. This category requires adjudication or further research. Consistent labeling ensures that your results reflect actual data quality rather than subjective interpretation.

• Use accurate supported for true claims with valid citations.
• Mark fabricated capability for invented products or features.
• Identify misattributed sources where real documents support wrong claims.
• Label confident statements without evidence as unsupported confident claims.

Human Coding and Agreement Metrics

Automated checks alone cannot catch all types of errors. Human coding provides the necessary nuance to distinguish between subtle hallucinations and genuine facts. For a robust verification study, assign two independent evaluators to code at least twenty percent of the outputs. This sample size allows you to measure the reliability of your classification system.

Calculate an agreement metric such as Cohen's kappa or Krippendorff's alpha to assess consistency between coders. A target score above 0.70 indicates that your definitions are clear and the labeling process is stable. Disagreements between evaluators highlight areas where definitions need refinement. Use these moments to clarify the rules for distinguishing between false attribution and misattribution.

Adjudicate conflicting labels through a third reviewer or a consensus discussion. This step ensures that the final dataset reflects a unified standard. Document the reasoning for any disputed claims. This record helps improve the protocol for future audits. Human oversight remains essential because AI systems can mimic authoritative tones even when stating falsehoods.

Scale the human effort based on your resources. A full study might require dual coding for all outputs, while a quick check might suffice with a smaller sample. The key is to establish a baseline of trust in your data. Without human verification, you risk accepting confident falsehoods as truth simply because they sound plausible.

• Deploy two independent coders for a minimum of 20% of outputs.
• Aim for an inter-rater agreement score above 0.70.
• Resolve disagreements through adjudication by a third party.
• Document the reasoning behind disputed classifications.

Checking the Sources Behind Claims

Verification requires checking three specific conditions for every cited source. First, confirm that the source actually exists and returns a valid connection. Second, ensure the source content matches the brand mentioned in the claim. Third, verify that the text within the source explicitly supports the specific assertion made by the AI. Missing any of these steps invalidates the claim.

Many AI systems generate URLs that look authentic but lead to dead pages or unrelated content. This phenomenon is known as fabricated source hallucination. In recent studies, high-salience brands faced higher rates of these errors because the models had more surface material to construct plausible but false completions. Always click every link and read the relevant section of the document.

Be aware that some models refuse to provide sources for low-salience entities while generating many for well-known ones. This behavior creates an engagement rate paradox where smaller companies appear less credible due to a lack of citations. When a source is missing, label the claim as accurate unsupported if you can verify it elsewhere, or as unsupported confident if you cannot.

Time sensitivity matters significantly in source verification. A document valid today might be outdated tomorrow, or a URL might break after a website redesign. Record the date and time of your check. Treat AI outputs as temporary snapshots of a changing digital environment. Do not assume that a source found today will remain accessible or accurate indefinitely.

• Verify that every cited URL returns a valid page.
• Confirm the source content belongs to the correct brand.
• Check that the text explicitly supports the specific claim.
• Record the timestamp of your source verification.

Recording Results for Comparison

Create a structured log for every verification run. Include fields for the platform name, model version, query class, and timestamp. This metadata allows you to identify patterns in errors across different systems. For instance, one platform might consistently hallucinate regulatory details while another struggles with product specifications.

Track the hallucination rate as the ratio of false claims to total claims. This metric provides a clear view of system reliability. Compare rates across different brand tiers to see if high-salience entities face higher risks. Recent data suggests that famous brands may receive more confident but false attributions than smaller companies under similar query conditions.

Store raw outputs and screenshots for auditability. These records serve as evidence if you need to dispute a claim or analyze a specific failure. Preserve the data in a secure format that protects privacy and intellectual property. This archive becomes valuable for longitudinal studies that track how model behaviors evolve over time.

Use the recorded data to refine your future queries and verification strategies. Identify which question types trigger the most errors and adjust your approach accordingly. This iterative process improves the quality of your audits. Systematic recording transforms isolated incidents into actionable insights about AI behavior.

• Log platform, model version, and query class for every run.
• Calculate the hallucination rate to measure system reliability.
• Archive raw outputs and screenshots for future analysis.
• Analyze trends to refine query strategies and verification methods.

AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.