How Often Is AI Wrong? Published Error Rates by Task, Sample Size, and Check Tier

In early October 2025, University of Sydney legal researcher Dr. Christopher Rudge opened a 237-page compliance review that Deloitte Australia delivered to the Department of Employment and Workplace Relations under an AU$440,000 contract (The Register, October 6, 2025). Inside sat more than a dozen nonexistent academic footnotes and an invented judicial quote attributed to Amato v Commonwealth. Deloitte uploaded a corrected report, agreed to repay its final contract installment (AP via CTV News, October 7, 2025), and disclosed staff had used Azure OpenAI GPT-4o to fill documentation gaps. Five months later, in March 2026, Chartered Professional Accountants Newfoundland and Labrador opened an investigation into a C$1.6 million Deloitte health-workforce report over fake citations (The Independent NL, March 7, 2026). Outside readers caught what internal teams missed.

The stakes hit ordinary desks every week. In a global survey by KPMG and the University of Melbourne (n > 48,000 respondents across 47 countries, November 2024 to January 2025, KPMG), 66% of surveyed workers relied on AI output without evaluating accuracy, and 56% of workers reported making mistakes due to unchecked AI output. Meanwhile, Damien Charlotin's tracker of court rulings addressing AI-fabricated authorities hit 2,241 cases worldwide by October 29, 2026, including 1,292 filings by self-represented litigants and 886 by lawyers (Charlotin Database).

As part of our program hub on the safe use of AI tools at work, this guide audits the major empirical benchmarks side by side with exact denominators, sponsors, and caveats. You leave with an expected-error-cost formula and the ed.cafe Task-by-Task AI Error Rate & Verification Tier Table.

Why asking for one AI hallucination rate hides the denominator

A single global AI accuracy rate does not exist because language model failure rates jump tenfold depending on whether the source text sits in the prompt, whether the test penalizes guessing, and who sponsored the benchmark. Every headline quoting one flat hallucination figure hides its denominator.

Asking "what is AI's error rate?" is like asking a hospital lab for the false-negative rate of "medical testing." A lab director will ask which assay you ordered: a rapid swab, a 48-hour blood culture, or a wrist X-ray. Blending all three into one hospital average misleads every patient. Where that lab analogy breaks down matters even more: chemical assays use fixed reagents with stable confidence intervals, whereas language models sample token probabilities across shifting prompts, context windows, and silent vendor updates.

Four confounding variables drive the gap between vendor slides and office reality:

• Supplied-source versus closed-book tasks: Pasting a short memo into the prompt keeps every fact in working context. Asking for a municipal code or case citation from training weights forces statistical reconstruction.
• Hard contradiction versus scope inflation: Automated judges catch invented names. Human readers get burned just as often when a summary drops preliminary or excluding overtime, turning a hedged finding into a universal rule.
• Abstention rules in scoring: Awarding zero points for "I do not know" trains models to bluff on obscure queries. Penalizing confident fabrications rewards models that flag missing evidence.
• Median ease versus p99 tail failures: Brainstorming tolerates a 15% error rate in draft ideas. Safety scripts or client tax memos fail at the p99 tail, where one silent defect in 100 runs erases months of saved hours.

Selection bias and unsourced folklore distort both sides of the debate. Transcription vendor blogs claim competitor AI accuracy sits "in the low-60% range" or quote meeting hallucination rates "around 70% to 80%" in unsourced posts (Entrepreneur; GoTranscript). Critics mirror that mistake when repeating the August 2025 MIT NANDA headline claiming "95% of corporate GenAI pilots get zero return." Wharton professor Kevin Werbach documented (Werbach critique) that MIT NANDA relied on 52 interviews and 153 conference survey responses, required rapid six-month P&L impact, and called its figures "directionally accurate" without publishing cohort data. Managers need denominated studies, not folklore.

Closed-passage summarization: why 0.7% to 3% became 1.8% to 24.2%

Closed-passage summarization delivers the lowest published fabrication rates in workplace AI, yet measured error rates span from 0.7% to 3% in early short-news benchmarks up to 1.8% to 24.2% in harder 2026 document benchmarks, while scope inflation hits 26% to 73% of scientific summaries. Pasting text into the prompt cuts invention sharply without making distortion vanish.

The standard benchmark for grounded summarization is Vectara's Hughes Hallucination Evaluation Model (HHEM) leaderboard (Vectara Leaderboard), which prompts models at temperature = 0 on supplied passages. Keep two caveats in view: Vectara sells enterprise retrieval software (making this a vendor-maintained benchmark), and dataset upgrades created cohort attrition that forbids plotting 2024 figures beside 2026 figures.

In Vectara's early public benchmark (n ~ 1,000 short news passages), top models clustered at a 0.7% to 3.0% hallucination rate. On Vectara's harder non-public HHEM-2.3 corpus of 7,700+ longer documents (updated May 11, 2026), error rates widened:

• Full 7,700+ document benchmark range: 1.8% error rate (best model) to 24.2% error rate (lowest model).
• Mid-2026 flagship interquartile range (7,700+ documents): gpt-5.4 and gemini-2.5-pro posted a 7.0% rate, gpt-5.5 hit 9.3%, claude-sonnet-4-6 hit 10.6%, claude-opus-4-6 hit 12.2%, and gemini-3-pro-preview hit 13.6% in that benchmark.

Measured rates moved from the 1% to 3% benchmark band into the 7% to 14% benchmark band because longer multi-entity documents create more chances to merge people or misattach numbers. Contradiction classifiers also miss scope inflation. Uwe Peters and Benjamin Chin-Yee studied n = 4,900 LLM summaries of scientific papers across 10 models in Royal Society Open Science (arXiv:2504.00025, March 2025): models stripped qualifiers (may, in this cohort) and overgeneralized findings in 26% to 73% of papers (OR = 4.85 versus humans), even when prompted for strict accuracy. Apple hit the consumer version in January 2025, suspending Apple Intelligence news notification digests after merged alerts generated false arrest headlines (RNZ, January 17, 2025).

Open-ended factual recall: what SimpleQA and Kalai et al. prove about guessing

Strip away the supplied document, and closed-book factual recall still produces double-digit fabrication rates in frontier models because binary accuracy leaderboards reward guessing over saying "I do not know."

OpenAI researchers Jason Wei and colleagues built SimpleQA in October 2024 to measure closed-book factuality (OpenAI SimpleQA; arXiv:2411.04368): n = 4,326 short, single-answer questions filtered against GPT-4 and graded correct, incorrect, or not attempted. Without web search, frontier models still miss or guess on large double-digit shares of SimpleQA prompts.

In September 2025, Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang proved why in "Why Language Models Hallucinate" (arXiv:2509.04664; OpenAI, September 5, 2025). In pretraining, rare singleton facts sit statistically close to noise. In post-training evaluation, standard leaderboards use binary 0/1 grading (+1 right, 0 wrong, 0 "I don't know"). Like an unpenalized multiple-choice test, binary scoring rewards bluffing over honest abstention.

OpenAI showed that split on SimpleQA (n = 4,326 questions from the GPT-5 System Card):

On raw accuracy, o4-mini edges ahead (24% correct rate vs 22% correct rate in 4,326 prompts). In office work, o4-mini fabricates on 75% of prompts, whereas gpt-5-thinking-mini abstains on 52% of prompts and cuts confident errors to a 26% rate.

Live AI search and news citation: why links do not prove the claim

Adding live web search reduces memory gaps, yet empirical audits show AI search engines misidentify or invent news citations in over 60% of queries and attach non-supporting web links to 56% of factually correct search answers. A blue link beside an AI paragraph never proves the linked page supports the sentence.

Three independent audits measure live search and sourcing errors:

1. Columbia Tow Center Study (March 6, 2025): Across 8 AI search tools and 200 publisher excerpts (n = 1,600 queries, CJR Tow Center), engines gave wrong or unsupported answers in over 60% of queries. Free Perplexity missed 37% of its 200 queries; Grok 3 missed 94% of queries and invented 154 error-page URLs in 200 trials. Paid tiers declined fewer prompts and logged higher confident-error rates than free tiers.
2. EBU and BBC Study (October 22, 2025): Journalists across 22 public broadcasters in 18 countries graded n > 3,000 news answers from ChatGPT, Copilot, Gemini, and Perplexity (VRT / EBU), finding a 45% issue rate across 3,000+ responses, a 31% sourcing-error rate, and a 20% major accuracy-error rate (Gemini hit a 76% issue rate in that sample).
3. Oumi / New York Times Study of Google AI Overviews (April 21, 2026): Across n = 4,326 SimpleQA queries (TechRepublic), Gemini 3-era snapshots hit a 91% accurate answer rate (9% wrong), up from 85% in Gemini 2-era snapshots, yet 56% of accurate answers in the 4,326 queries cited ungrounded URLs that did not support the claim. (Google contested extrapolating SimpleQA trivia to daily traffic, noting standalone Gemini 3 without search erred on 28% of queries.)

Checking whether a URL loads misses the 56% ungrounded-citation rate. See our guide on checking every source in an AI-assisted report for line-by-line claim verification.

Domain RAG, clinical scribes, and spreadsheet math: vertical tools still misfire

Specialized retrieval tools, medical scribes, and spreadsheet assistants still carry steep task error rates: peer-reviewed audits measure 17% to 34% hallucinations in commercial legal RAG software, 11.5% hallucinations plus 18% omissions in clinical scribe records, and 53.3% logic defects in AI-written safety calculation scripts.

Stanford RegLab researchers Varun Magesh and colleagues tested commercial legal retrieval-augmented generation (RAG) tools marketed as "hallucination-free" in the Journal of Empirical Legal Studies (June 2025, 22(2), 216-242, DOI: 10.1111/jels.12413; Stanford HAI). Across n = 200+ pre-registered legal queries, Lexis+ AI and Ask Practical Law AI hallucinated or misgrounded authorities in over 17% of queries, while Westlaw AI-Assisted Research erred in over 33% of queries (>34% in Stanford HAI's release table) by citing dissents as holdings, wrong courts, or superseded statutes.

Clinical audio scribes show a similar gap. At UC Davis Health, physicians graded n = 356 ambient AI visit records sampled from 7,545 encounters (JMIR Medical Informatics, April 17, 2026, JMIR Med Inform), finding an 18.0% omission rate, 11.5% hallucination rate, 9.3% accidental-inclusion rate, and 5.3% severe clinical-risk rate across 356 records (94.7% of records had no severe-risk error). At the audio layer, Koenecke et al. (arXiv:2402.08021, 2024) found ~1% of Whisper transcripts invented whole sentences (38% of cases harmful), and an AP investigation (AP via KMPH, October 26, 2024) reported a Whisper-based hospital scribe used in ~7 million visits auto-erased source audio. See our guides on case managers using AI for case notes and AI notetakers in meetings.

Multi-step formulas and safety math fail hardest under single-shot generation:

• Excel workbooks (SpreadsheetBench): Across n = 912 Excel workflows (arXiv:2406.14991, 2024), single-turn models failed on 75% to 85% of tasks (15% to 25% strict pass rates) from merged-header offsets.
• Perturbed math (GSM-Symbolic): Across 50 symbolic variations per template (arXiv:2410.05229, 2024), minor number swaps or one distractor clause triggered steep accuracy drops.
• OSHA safety scripts: Across n = 450 LLM Python scripts for OSHA 29 CFR 1926 rules (Uddin, April 2026, arXiv:2604.12311), 85.3% of scripts ran, yet only 46.7% of scripts had correct OSHA logic (53.3% failure rate), and 45.3% of runnable scripts failed silently.

How to calculate expected error cost before automating a workflow

An AI workflow pays off only when net labor saved after checking exceeds the expected cost of residual defects escaping externally. The ed.cafe Expected Error Cost Formula prices that breakeven:

Expected Error Cost Formula: Net Monthly Value = N * [(T_saved - T_check) * W - (p_residual * C_miss)]

Here N is monthly volume, T_saved is gross hours saved per item, T_check is human check hours, W is hourly wage, p_residual is the defect rate after review, and C_miss is cost per escaped error. Test two workflows at a hypothetical 35-person Boston accounting firm (assumptions stated):

Workflow A: Internal digest of a supplied 6-page vendor PDF. Set N = 200 digests/month, T_saved = 0.25 hr, T_check = 0.05 hr (3-minute skim), W = $60/hr, and p_raw = 7.0% (Vectara HHEM-2.3, 7,700+ docs). Catching 70% of errors in review leaves a 2.1% residual rate; internal clarification costs C_miss = $40. Labor saved $2,400 minus defect cost $168 = +$2,232 per month.

Workflow B: Client citation memos via live AI search. Set N = 40 memos/month, T_saved = 1.00 hr, W = $85/hr, and p_raw = 35% (Stanford RegLab 17% to 34% legal RAG rate; Tow Center 37%+ search error rate). Microsoft and Carnegie Mellon (n = 319 workers, The Register) link AI confidence to lower critical-thinking effort: a 6-minute prose skim (T_check = 0.10 hr) without opening links catches just 20% of misgrounded cases (28% residual rate). Price an escaped client error at C_miss = $1,800 in write-offs and credits (above Stanford/BetterUp's survey-based $186/month internal "workslop" estimate across n = 1,150 US workers, Axios, and below a $116,315.09 Sixth Circuit sanction in March 2026, JD Supra):

• 6-minute skim (T_check = 0.10 hr, 28% residual rate): $3,060 saved minus $20,160 defect loss = -$17,100 per month.
• 27-minute URL check (T_check = 0.45 hr, catching 98% of errors in review for a 0.7% residual rate): $1,870 saved minus $504 defect loss = +$1,366 per month.

Measure your own p_raw with our 50-example spreadsheet method to test an AI tool and score drafts with our 12-point rubric for AI work samples.

How to match every office task to its human verification tier

Every workplace AI task maps to one of four human check tiers based on its empirical error base rate and external harm severity. The ed.cafe Task-by-Task AI Error Rate & Verification Tier Table replaces vague "review AI output" rules with concrete checks across Tiers 1 to 4. We drill these tiers in our 75-minute adult hallucination lab and inside No Worker Left Behind's pre-launch Circle #1 cohort in Boston (30 free chairs for the Google AI Professional Certificate via the Massachusetts AI Hub, closing December 1, 2026).

THE ED.CAFE TASK-BY-TASK AI ERROR RATE & VERIFICATION TIER TABLE (Q4 2026)
Publisher: ed.cafe by No Worker Left Behind (501(c)(3), Boston, MA)

FOUR-TIER VERIFICATION RULE
- TIER 1 (90-Sec Skim): Read numbers, dates, deadlines, and "not/except" aloud.
- TIER 2 (Source-vs-Claim): Ctrl+F figures, names, and hedged qualifiers in source file.
- TIER 3 (Primary Lookup / Native Math): Click every URL, match exact proving sentence,
  verify quotes verbatim, keep source audio, and use native Excel/Sheets formulas.
- TIER 4 (Do Not Delegate): Use certified CFR tables, code-enforced payout caps, and
  human corroboration with HR/legal review.

--- COPY-PASTE CSV TABLE (10 WORKPLACE TASKS WITH SAMPLE SIZES) ---
Task,Primary_Study,Sample_n,Measured_Error_Range,Tier,Mandatory_Human_Check
1_Email_Polish,Author bullets,N/A,<3% drift on short text,Tier 1,Read numbers dates and negations aloud
2_Doc_Summary,Vectara HHEM-2.3 (May 2026),7700+ docs,1.8% to 24.2% unsupported,Tier 2,Ctrl+F figures and entities in source PDF
3_Science_Summary,Peters & Chin-Yee (Mar 2025),4900 papers,26% to 73% scope inflation,Tier 2,Restore hedged scope words (may / cohort)
4_Meeting_Notes,UC Davis JMIR (2026) + Whisper,356 records,11.5% hallucination + 18% omission,Tier 2/3,Keep audio and verify owners/numbers in 24h
5_Closed_Fact_QA,OpenAI SimpleQA (Sep 2025),4326 prompts,26% (52% abstain) to 75% wrong,Tier 3,Prompt to abstain if unsure; check primary DB
6_AI_News_Links,CJR Tow Center (Mar 2025),1600 queries,>60% wrong or unsupported,Tier 3,Open every URL and match proving sentence
7_Search_Overviews,Oumi / NYT (Apr 2026),4326 queries,9% wrong + 56% ungrounded URLs,Tier 3,Confirm linked page actually states the claim
8_Legal_Policy_RAG,Stanford RegLab JELS (Jun 2025),200+ queries,>17% (Lexis+) to >33% (Westlaw),Tier 3,Read full opinion/statute and verify good law
9_Spreadsheet_Math,SpreadsheetBench (Jun 2024),912 tasks,75% to 85% single-turn failure,Tier 3,Require native Excel formulas + control totals
10_OSHA_Safety_Code,Uddin CSU (Apr 2026),450 scripts,53.3% wrong logic (45.3% silent),Tier 4,Do not use raw LLM math; use printed CFR tables

Where this stops being true

Four boundaries limit how far you can generalize the benchmarks in this guide:

• Dataset upgrades break trend lines: Vectara's May 2026 HHEM-2.3 benchmark (n = 7,700+ articles) replaced its older short-passage set; higher 2026 rates reflect harder documents, not weaker models.
• 2024 vertical benchmarks await replication: Stanford RegLab's legal RAG study (n = 200+ queries, published June 2025) tested 2024 versions of Lexis+ AI and Westlaw; peer-reviewed replications on 2026 builds have not yet appeared.
• Code execution changes raw math failure rates: Having a model write an inspectable Excel formula overcomes raw token-math brittleness (GSM-Symbolic), provided a human checks domain logic (53.3% failure rate in Uddin's n = 450 OSHA scripts).
• Professional licensing overrides cost math: Attorneys, clinicians, CPAs, and safety engineers hold non-delegable sign-off duties. Where employment laws or your one-page small-business AI use policy restrict data use, counsel and compliance officers govern.

Questions people ask

Key terms

• Base rate: Baseline share of prompts in a task where an AI errs before human review.
• Closed-passage summarization: Condensing text pasted in the prompt rather than recalling facts from weights.
• Abstention calibration: Prompting or training a model to say "I do not know" when evidence is missing.
• Ungrounded citation: A working URL on an AI claim whose page never proves that claim.
• Misgrounding: Citing a real case, statute, or policy while misstating its holding, date, or scope.
• Silent failure: Runnable AI code or formulas executing cleanly while computing wrong numbers.
• Residual error rate (p_residual): Share of defective AI outputs slipping past human review into delivery.

Sources

1. The Register, "Deloitte to refund Australian government after using AI in AU$440k report" (October 6, 2025): The Register
2. Associated Press via CTV News, "Deloitte to partially refund Australian government for report" (October 7, 2025): AP / CTV News
3. The Independent NL, "Accounting watchdog to investigate Deloitte over fake citations in $1.6M report" (March 7, 2026): The Independent NL
4. KPMG & University of Melbourne, "Trust, attitudes and use of AI: Global study 2025 (n > 48,000)" (April 2025): KPMG Release
5. Damien Charlotin, "AI Hallucination Cases Database (2,241 decisions through October 29, 2026)" (October 2026): Charlotin Database
6. Vectara, "HHEM-2.3 Summarization Leaderboard (n = 7,700+ articles)" (May 11, 2026): Vectara GitHub
7. Uwe Peters & Benjamin Chin-Yee, "Generalization bias in LLM summarization (n = 4,900)" Royal Society Open Science (March 2025): arXiv:2504.00025
8. Jason Wei et al., OpenAI, "Introducing SimpleQA (n = 4,326 questions)" (October 30, 2024): OpenAI SimpleQA
9. Adam Tauman Kalai et al., "Why Language Models Hallucinate" (September 5, 2025): OpenAI and arXiv:2509.04664
10. Columbia Journalism Review / Tow Center, "We Compared Eight AI Search Engines (n = 1,600 queries)" (March 6, 2025): CJR Tow Center
11. EBU & BBC, "News Integrity in AI Assistants (n > 3,000 responses)" (October 22, 2025): VRT / EBU
12. TechRepublic (Oumi / NYT study of Google AI Overviews, n = 4,326), "AI Overviews Analysis" (April 21, 2026): TechRepublic
13. Varun Magesh et al., Stanford RegLab, "Hallucination-Free? Assessing AI Legal Research Tools" JELS 22(2), 216-242 (June 2025): DOI 10.1111/jels.12413
14. UC Davis Health, "Ambient AI Scribe Notes (n = 356)" JMIR Medical Informatics (April 17, 2026): JMIR PDF
15. Allison Koenecke et al., "Careless Whisper" (February 2024): arXiv:2402.08021
16. Zeyao Ma et al., "SpreadsheetBench (n = 912 tasks)" (June 2024): arXiv:2406.14991
17. Iman Mirzadeh et al., Apple, "GSM-Symbolic" (October 2024): arXiv:2410.05229
18. Uddin, Colorado State University, "LLM Code Generation for OSHA 29 CFR 1926 (n = 450)" (April 2026): arXiv:2604.12311
19. Kevin Werbach, "Critique of MIT NANDA GenAI Report" (August 2025): Critique Mirror