keepcalculator

06 Numbers and Powers

How to Fact-Check Artificial Intelligence Claims

Card No. 06·77623 · Updated · Standard formula, shown below

Fact-checking claims about artificial intelligence helps you separate demonstrated capabilities from marketing, speculation, and misunderstandings. Start with the exact claim and its original source; then look for independent evidence, check what was actually tested, and report any uncertainty clearly.

What You’ll Need

  • The claim in its original wording, plus a link, screenshot, or citation to where it appeared.
  • Access to reliable sources such as research papers, official documentation, reputable reporting, and evaluation results.
  • A way to record sources, dates, methods, and caveats. A simple document or notes app is enough.
  • For technical claims, basic context about the system, task, version, and conditions being discussed. If those details are missing, note that as a limitation.

Step-by-Step

  1. Write down the claim precisely. Preserve the original wording before interpreting it. “The model can summarize documents” is different from “the model accurately summarizes every document.” Separate compound statements into individual claims, and identify whether each is about capability, performance, safety, cost, adoption, or future potential. Claims about the future cannot be verified in the same way as a completed experiment.
  2. Find the earliest available source. Follow links and citations back from a social post, news story, or presentation to the study, product documentation, dataset, or person making the claim. Check whether the source actually supports the wording being repeated. If you cannot locate an original source, say so rather than treating repetition as confirmation.
  3. Define what would count as evidence. Turn vague language into a testable question. For “more accurate,” ask: accurate on which task, compared with what, and measured how? For “understands,” identify the observable behavior meant by that term. For a claim that a system is “safe,” ask which risks were assessed and under what conditions. A claim that has no clear test may be opinion, interpretation, or prediction rather than a verifiable fact.
  4. Check dates and system details. Record when the evidence was produced and which model, software version, settings, tools, and data were involved. AI systems change, and results from one version or setting do not automatically apply to another. Look for the evaluation date as well as the publication date; an older test may no longer describe a current system.
  5. Read the evidence, not just its headline. For a research paper, inspect the abstract, methods, results, limitations, and any supplementary material relevant to the claim. For a benchmark, check which tasks and scoring rules it uses. For a company announcement, distinguish published measurements from promotional wording. Note whether the result comes from a controlled test, a small demonstration, a survey, or routine use.
  6. Check the comparison is fair. Ask whether the systems were given comparable prompts, tools, time, data, and opportunities to retry. Check whether the test data were separate from training data, when that information is available. See if the result is based on enough examples and whether the scoring method matches the task. A high score on a narrow benchmark does not by itself establish broad real-world ability.
  7. Look for independent confirmation and counterevidence. Search for replication, evaluations by independent researchers, corrections, and credible criticism. Prefer sources with transparent methods and relevant expertise, but do not accept a claim solely because a source is prestigious. One confirming example proves that something happened once; it does not establish how often it happens. One failure also does not establish that a system always fails.
  8. Check numbers and context. Verify the denominator, sample size, time period, and baseline behind percentages or rankings. Ask whether a reported improvement is absolute or relative, and whether the difference matters for the claim. Check if the result is averaged across cases that vary substantially. Do not compare scores from different tests as if they were measured on one common scale unless the methods support that comparison.
  9. Separate what the system did from what people infer. A fluent answer does not prove that a system understands, has intentions, or independently verified its statements. A generated citation should be checked against the cited source. A demonstration shows performance in its displayed conditions, not necessarily reliability in other settings. Describe observed behavior without adding unsupported explanations about how or why it happened.
  10. Write a calibrated verdict. Use a conclusion such as “supported,” “partly supported,” “misleading,” “unsupported by the available evidence,” or “not yet verifiable.” Explain the key reason in plain language, link the best evidence, and include important limits. If evidence is mixed or incomplete, say what is known, what remains uncertain, and what new evidence could change the conclusion.

Common Mistakes

  • Checking a paraphrase instead of the original: A post may remove a qualification or turn a limited result into a universal claim. Compare the wording with the source.
  • Treating a benchmark as proof of general ability: Benchmarks cover selected tasks and can have limitations, including overlap with training material. Explain the test’s scope.
  • Trusting a fluent answer or citation: Confident phrasing is not evidence. Open the cited paper or dataset and confirm it says what the answer claims.
  • Confusing correlation with causation: A study may show that two things occur together without establishing that AI caused one of them. Look for the research design and stated limitations.
  • Ignoring changes over time: A finding about an earlier version may not describe a newer one. Match evidence to the relevant date and system.
  • Using a false binary: A claim may be partly true or true only under certain conditions. Avoid forcing a yes-or-no verdict when the evidence is qualified.

Tips

  • Search key phrases from the claim alongside terms such as “evaluation,” “methods,” “limitations,” or “replication.”
  • Keep a short evidence log: source, date, system tested, method, result, and caveat. This makes your conclusion easier to review.
  • Prefer primary documents for what was measured, then use independent analysis to understand context and limitations.
  • Distinguish the existence of a feature from how well it works. Availability does not prove accuracy, reliability, or widespread use.
  • When describing a result, retain its scope: name the task, conditions, and date rather than saying simply that “AI can” do something.

Troubleshooting

The original source is unavailable: Search the exact phrase and the speaker or organization. Check archived copies or cited documents where appropriate. If the source remains inaccessible, explain that the claim could not be independently checked and avoid presenting it as established fact.

Sources disagree: Compare their dates, definitions, test conditions, and methods before deciding whether they truly conflict. A newer result may use a different task, or both findings may be valid within different limits. Report the disagreement and explain what distinguishes the evidence.

The claim uses vague terms such as “intelligent” or “human-level”: Ask the claimant to define the term, or state the measurable interpretation you are checking. If there is no shared, testable definition, label the claim ambiguous rather than assigning a confident verdict.

You cannot judge the technical method: Look for an independent review by qualified researchers, a transparent methods section, or a clear explanation of the test. Do not infer that a claim is true or false just because the technical details are difficult; state which parts you could and could not verify.

The evidence is too limited for a firm answer: Narrow the conclusion to what the evidence does establish, such as a result on one test or in one demonstration. Identify the missing information and say what further testing would be needed.

Frequently Asked Questions

Can an AI system fact-check itself?
It can help locate claims or suggest sources, but its output needs independent verification. Check every important quotation, link, and factual statement against the original material.

Are peer-reviewed studies always reliable?
No. Peer review is one quality check, not a guarantee. Review the methods, sample, limitations, later corrections, and whether independent work supports the finding.

Does a high benchmark score prove a system is dependable?
No. It shows performance under that benchmark’s conditions. It does not establish performance on different tasks, users, data, or real-world situations.

How should I check a claim about a model’s latest version?
Find documentation or evaluations that identify the version and test date. If those details are absent, explain that the evidence may not apply to the current version.

What if a claim cannot be tested yet?
Label it as a prediction or opinion, not a verified fact. Record what future evidence or measurable outcome could confirm or disconfirm it.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →