Three public failures between 2024 and 2025 show why AI-assisted work never breaks in just one way. On February 14, 2024, British Columbia's Civil Resolution Tribunal (Moffatt v. Air Canada, 2024 BCCRT 149) ordered Air Canada to pay compensation after its website chatbot invented a 90-day retroactive bereavement refund rule that contradicted the airline's real tariff (The Guardian, February 16, 2024). On January 17, 2025, Apple suspended Apple Intelligence news notification summaries after merged alerts fabricated BBC headlines, including a false arrest report (RNZ, January 17, 2025). Then in May 2025, reporters checking the White House MAHA report found at least seven cited medical studies among 500+ footnotes that had never been published (CBS News, May 30, 2025).
Reading an AI draft from top to bottom like a human memo fails because polished grammar hides five distinct mechanical failure modes. Before you send AI-assisted work, match your check to the output type: re-total numbers in a real spreadsheet, search exact strings for every direct quote, open every policy or legal citation in a browser tab, audit summaries for dropped caveats, and test formulas or SQL on a five-row known slice using the ed.cafe matrix below.
Most workplace mistakes happen right at that handoff moment. In the global study by KPMG and the University of Melbourne (n > 48,000 respondents across 47 countries, surveyed November 2024 to January 2025, KPMG, April 2025), 66% of surveyed employees relied on AI output without evaluating accuracy, and 56% of workers reported making work mistakes due to AI. A 2025 study by Microsoft Research and Carnegie Mellon University (n = 319 knowledge workers across 936 task examples, Microsoft Research, CHI 2025) showed why: higher confidence in AI correlates directly with lower critical-thinking effort during review.
Sitting within our practical hub on AI safety at work, this guide turns the U.S. Department of Labor's "evaluating outputs" competency into a fast desk routine. While our 12-point rubric for AI work samples scores finished training portfolios, this guide gives you the ed.cafe 5-Column Pre-Send Verification Matrix by Output Type.
Why proofreading AI drafts like human writing misses mechanical defects
Proofreading an AI draft from top to bottom catches awkward phrasing while missing nearly every math, fact, and code defect. Policies tell staff to "keep a human in the loop," yet rarely state what the eye must test.
For decades, messy prose warned reviewers to check the math when a rushed colleague guessed a midnight figure. Language models broke that link between smooth prose and true records, as explained in why ChatGPT makes things up in plain English.
Think of reviewing AI output like a kitchen safety check before dinner service. Shiny stainless counters prove zero food safety; a line lead uses a probe thermometer for poultry, a test strip for sanitizer, and an invoice check at the dock. Where that analogy breaks down matters too: thermometers measure fixed laws, whereas one AI email can nail tone in line one, drop a minus sign in line two, and invent a software library in line three.
Sort every draft into five output buckets before checking style: (1) numbers and tables, (2) direct quotes, (3) legal, academic, and policy citations, (4) long-document summaries, and (5) scripts, formulas, and SQL.
Output types 1 and 2: numbers, spreadsheets, and direct quotations
Numbers and direct quotes require deterministic matching rather than eyeball checks because chat models predict plausible digits and quotation marks without verifying cells or transcripts.
Output type 1: Numbers, percentages, and spreadsheet tables
Without code execution, text models predict digits that look arithmetic rather than summing rows. Even when chat tools run Python on a CSV, three quiet bugs recur:
- Silent row drops: Scripts skip blank strings,
"$1,200"text cells, or trailing spaces, shrinking totals silently. - Swapped denominators: Margin gets swapped for markup, or survey percentages lose their sample size (
n). - Fiscal-quarter drift: October-to-December rows land in Q4 even when your fiscal year starts July 1.
The 90-second number check: Paste every AI table into Excel or Sheets and write fresh =SUM() and percentage formulas beside it. Match COUNTA row counts to your raw CSV, check 3 raw rows, and attach n to every percentage.
Output type 2: Direct quotations
Quotation marks promise exact words in exact order. Models break that promise by wrapping compressed paraphrases in quotes or inventing quotes inside real files, as in July 2025 when Deloitte Australia delivered a review under an AU$440,000 contract containing a fake judicial quote in Amato v Commonwealth and repaid its final contract installment in October 2025 (The Register, October 6, 2025).
The 60-second quote check: Copy five to seven unique words inside every AI quote and press Ctrl+F (Cmd+F on Mac) in the primary PDF, page, or transcript. Zero hits mean replace with the true sentence or drop the quotes.
Output types 3 and 4: policy citations and executive summaries
Citations and executive summaries fail even when links open and sentences sound balanced, because models attach ungrounded URLs and strip cautious qualifiers from source files.
Output type 3: Legal, academic, and policy citations
Clicking a hyperlink only to see if a webpage loads misses over half of AI sourcing errors:
- Legal RAG tools (
n > 200queries): Stanford RegLab and HAI tested commercial legal AI tools in May 2024 and found hallucination rates above 17% in Lexis+ AI and Ask Practical Law AI, and above 33% in Westlaw AI-Assisted Research (Stanford HAI, May 23, 2024). Damien Charlotin's tracker reached2,241rulings globally (1,514US;1,292self-represented) as of October 29, 2026 (Charlotin Database). - Ungrounded search links (
n = 4,326searches): An April 2026 Oumi audit of Google AI Overviews onSimpleQAfor The New York Times found 91% accurate answers, yet 56% of accurate answers cited pages that failed to support the claim (TechRepublic, April 21, 2026). - Broken municipal rules: NYC shut down its MyCity business chatbot in early 2026 after it told employers they could keep worker tips and refuse housing vouchers (The Markup, January 30, 2026).
The 2-minute citation check: Open every cited link, statute, or HR rule in your own tab and confirm: (1) it exists, (2) the cited paragraph backs your claim, and (3) no amendment superseded the date. For formal reports, see our five-step guide to checking every source and disclosing AI use.
Output type 4: Executive summaries of long documents
Summarization fails mainly through scope inflation. Uwe Peters and Benjamin Chin-Yee audited n = 4,900 LLM science summaries across 10 models in Royal Society Open Science (arXiv:2504.00025, 2025) and found overgeneralized conclusions in 26% to 73% of summaries (odds ratio = 4.85 vs humans). On contradiction checks, Vectara's May 2026 HHEM-2.3 benchmark (n = 7,700+ long documents at temperature = 0) logged 1.8% to 24.2% error rates (Vectara Leaderboard).
The 3-minute "Scope-Number-Exception" check: (1) Circle all, always, never, and company-wide in the draft and verify the source did not limit the finding to one pilot; (2) check "Exclusions" headings for deleted carve-outs; and (3) confirm two speakers or alerts were not merged into one claim.
Output type 5: scripts, spreadsheet formulas, and SQL queries
AI-generated Excel formulas, SQL queries, and Python scripts fail most dangerously when they execute without syntax errors while returning wrong rows or pulling nonexistent software packages:
- Truncated ranges and join multiplication:
=SUMIFS(C2:C100, A2:A100, "East")runs cleanly on148rows while dropping rows101to148. In SQL, aLEFT JOINagainst multi-rowshipmentsduplicates revenue for split shipments. - Invented packages ("slopsquatting"): In USENIX Security 2025 (arXiv:2406.10279), Joseph Spracklen et al. tested 16 code LLMs across
n = 576,000Python and JavaScript samples; 19.7% of package installs (21.7%open-source,5.2%+commercial across576,000samples) named nonexistent packages that attackers can register on PyPI or npm.
The 3-minute formula and script check: (1) Run on a 5-row dummy table with a hand-calculated answer (1 blank/NULL, 1 boundary date, 1 duplicate ID); (2) confirm cell ranges hit the bottom row and check COUNT(DISTINCT id) around SQL JOINs; and (3) verify every pip or npm library on PyPI or npm before installing.
Summary comparison and worked 11-minute check on a Monday packet
Matching the verification step to the output format catches the exact errors prose skimming misses:
| Output Type | Dominant Failure Mode | Benchmark Signal | Skimming Mistake | Mechanical Pre-Send Check |
|---|---|---|---|---|
| 1. Numbers & Tables | Dropped text/null rows, wrong denominator. | 56% of workers report AI mistakes (n > 48,000 survey, 2025). | Eyeballing plausibility. | Re-sum in Sheets, match COUNTA, check 3 rows + n. |
| 2. Direct Quotes | Paraphrase in quotes, invented quote. | Deloitte AU$440k fake court quote (Oct 2025). | "Sounds like them." | Ctrl+F (Cmd+F) 5 words in primary file. |
| 3. Citations & Rules | Live URL fails to back claim, expired rule. | 56% of right AI Overviews ungrounded (n = 4,326, 2026). | Checking HTTP 200 only. | Open tab, read paragraph, check active date. |
| 4. Long Summaries | Scope inflation, dropped exclusions. | 26% to 73% overgeneralized (n = 4,900 summaries, 2025). | Reading only for flow. | Audit all/always, exclusions, speaker IDs. |
| 5. Formulas & Code | Short B2:B100 range, JOIN doubling, fake pkg. | 19.7% fake packages (n = 576,000 samples, 2025). | "No error" = right math. | Test 5-row slice (NULL + dup) + check PyPI/npm. |
Worked time-budget example: checking a Monday packet in 11 minutes
Suppose an analyst uses an approved workplace AI tool to draft a Monday packet (assumptions: 1 KPI table of 8 metrics, 2 survey quotes, 3 grant rules, a 1-page summary of a 28-page RFP, and 1 SUMIFS formula). Unchecked, it creates workslop that shifts two hours of cleanup onto teammates. Manual drafting takes 75 minutes; running the ed.cafe 5-Column Pre-Send Verification Matrix by Output Type takes 11 minutes:
- KPI table (
2.0 min): Paste into Sheets, run=SUM(), matchCOUNTA(412rows), check3rows (catches 1 skipped "$1,450" text cell). - Two client quotes (
1.5 min):Cmd+Ffive words per quote in survey CSV (restores 1 paraphrased quote). - Three grant citations (
4.5 min): Open agency PDF (confirms 2 rules, catches 1 outdated FY2024 cap). - One-page RFP summary (
2.0 min): Check RFP "Exclusions" (restores a dropped 30-day subcontract notice rule). - One
SUMIFSformula (1.0 min): Confirm range hits row412and test on a 5-row slice.
Copy-paste artifact: The ed.cafe 5-Column Pre-Send Verification Matrix by Output Type
Pin the ed.cafe 5-Column Pre-Send Verification Matrix by Output Type beside your inbox or team wiki.
=======================================================================================
THE ED.CAFE 5-COLUMN PRE-SEND VERIFICATION MATRIX BY OUTPUT TYPE
Free workplace verification card from ed.cafe (No Worker Left Behind)
Rule: Never proofread AI work by skimming top-to-bottom. Check by output bucket.
=======================================================================================
| 1. OUTPUT TYPE | 2. PRIMARY FAILURE MODE | 3. EXACT MECHANICAL CHECK | 4. TIME BUDGET | 5. STOP-SHIP RED FLAG |
|---------------------|-------------------------|----------------------------------|----------------|--------------------------------|
| NUMBERS & TABLES | Silent dropped rows, | Re-total in Excel/Sheets with | 90 sec per | Row count (COUNTA) mismatches |
| | swapped denominator, | =SUM(); match COUNTA; verify 3 | table | raw export, or any % lacks its |
| | wrong fiscal quarter. | raw rows + attach n to every %. | | real denominator / date window.|
|---------------------|-------------------------|----------------------------------|----------------|--------------------------------|
| DIRECT QUOTATIONS | Paraphrase wrapped in | Copy 5-7 unique words inside | 30-60 sec per | Ctrl+F returns 0 matches in |
| | quotes, or invented | quotes; run Ctrl+F / Cmd+F in | quote | the original transcript, PDF, |
| | quote in a real file. | the primary source document. | | or court/policy record. |
|---------------------|-------------------------|----------------------------------|----------------|--------------------------------|
| POLICY, LEGAL & | Live link fails to back | Open every URL/statute in your | 90-120 sec per | Page loads (HTTP 200) without |
| ACADEMIC CITATIONS | claim, ghost title, or | own tab; confirm title, exact | citation | stating your claim, or statute |
| | superseded rule date. | paragraph support & active date. | | / guidance has been amended. |
|---------------------|-------------------------|----------------------------------|----------------|--------------------------------|
| EXECUTIVE SUMMARIES | Scope inflation ("all" | Circle "all/always/never" in | 2-3 min per | Cautious pilot language turned |
| OF LONG DOCUMENTS | vs "pilot"), dropped | draft; compare against source | 1-page summary | universal, or an "Exclusions" |
| | exceptions, merged IDs. | Exclusions/Limitations section. | | clause vanished from the memo. |
|---------------------|-------------------------|----------------------------------|----------------|--------------------------------|
| SCRIPTS, FORMULAS | Truncated cell range, | Run on a 5-row hand-calculated | 2-3 min per | Untested JOIN changes unique |
| & SQL QUERIES | SQL JOIN row doubling, | slice (with 1 NULL + 1 dup ID); | formula/query | ID count, or AI suggests a new |
| | fake PyPI/npm package. | verify packages on PyPI/npm. | | package not verified on PyPI. |
---------------------------------------------------------------------------------------
OPTIONAL 1-LINE HANDOFF FOOTER (PASTE AT BOTTOM OF INTERNAL DRAFTS):
"AI-assist check complete: [Drafted/Summarized with <Tool> | Verified: math in Sheets,
<X> quotes via Ctrl+F, <Y> primary links opened, exclusions checked by <Initials>]."
=======================================================================================Where this stops being true
Desk checks catch everyday hallucinations, yet four clear limits govern the ed.cafe 5-Column Pre-Send Verification Matrix by Output Type:
- Spot-checking 3 rows never suffices for payroll, taxes, or audited accounting: Binding financial records require full line-by-line deterministic reconciliation in accounting or payroll software, never AI prose tables plus spot checks.
- 5-row toy slices do not replace production security audits: Five synthetic rows catch business-logic and
JOINbugs for internal analysis, yet never replace code review, SQL-injection parameterization, or secret scanning on live software. - Regulated legal, clinical, and physical-safety domains require licensed sign-off: This guide provides general workplace operational practices, not legal, clinical, or licensed engineering advice. Court filings require full citator checks, clinical notes require licensed clinician review, and OSHA electrical or lockout math must come from primary standards.
- Benchmark figures reflect specific model vintages: Treat studies like Stanford HAI (May 2024) and Peters and Chin-Yee (2025) as proof of recurring failure modes rather than fixed error constants. Learners practicing these checks can also explore our free Circle #1 starting page, where applications for 30 Massachusetts seats close December 1, 2026.
Questions people ask
How do I fact-check ChatGPT output quickly before sending an email?
Skip re-reading for grammar first and highlight only checkable claims: numbers, dates, direct quotes, policy rules, and links. Spend two to five minutes verifying each item in its primary source (=SUM() in Sheets for math, Ctrl+F for quotes, and a live browser tab for rules), then polish tone last.
Can I paste an AI draft into a second AI tool to fact-check it?
A second model in a fresh chat window can spot scope inflation when you supply the primary text alongside the draft, yet two language models share similar blind spots on closed-book facts, URLs, and arithmetic. Always verify links, quotes, and spreadsheet sums in deterministic non-AI tools.
Why do AI summaries of real PDFs still mislead coworkers?
Even when given the real PDF, models compress conditional findings into broad takeaways. Across n = 4,900 scientific summaries audited by Peters and Chin-Yee in 2025, models dropped qualifiers and overgeneralized conclusions in 26% to 73% of summaries (arXiv:2504.00025). Always compare universal words (all, always, never) against the PDF's exclusions section.
How can I tell if an Excel formula or SQL query written by AI is wrong?
Never trust a formula just because it avoids #VALUE! errors. Click the formula bar to confirm cell ranges reach the final row of your table, check row counts before and after any JOIN, and test the formula on a five-row slice containing one blank cell and one duplicate ID calculated by hand.
Should I tell my manager or client when I verified an AI-assisted draft?
Yes, a short verification stamp builds trust and separates checked analysis from raw output. Adding one plain line stating what AI drafted and which specific checks you completed on spreadsheet math, quotes, and primary links answers reviewer questions upfront and models transparent practice.
Key terms
- Output-type verification: Matching your human fact-check step to the distinct failure mode of numbers, quotes, citations, summaries, or code instead of skimming prose.
- Scope inflation: When an AI summary strips qualifiers ("in one pilot," "may") from a real file and states a narrow finding as a universal rule.
- Ungrounded citation: A live web link attached to an AI sentence whose underlying page text fails to state or support the claim.
- Package hallucination (slopsquatting): When a code model suggests installing a nonexistent software library name that attackers can register with malicious code.
- Task stewardship: Described in Microsoft Research and CMU's 2025 study (
n = 319workers), the shift from blank-page drafting to auditing and verifying AI outputs. - Synthetic toy slice: A 5-row test table with a hand-calculated answer (including blank and duplicate edge cases) used to verify formulas or SQL before full runs.
Sources
- KPMG International and University of Melbourne, "Trust, Attitudes and Use of Artificial Intelligence: A Global Study 2025" (April 2025,
n > 48,000): KPMG Global Release - Hao-Ping (Hank) Lee et al. (Microsoft Research & Carnegie Mellon University), "The Impact of Generative AI on Critical Thinking," ACM CHI '25 (2025,
n = 319workers): Microsoft Research Publication - Uwe Peters and Benjamin Chin-Yee, "Generalization Bias in Large Language Model Summarization of Scientific Research," Royal Society Open Science (2025,
n = 4,900summaries): arXiv:2504.00025v1 - Joseph Spracklen et al., "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs," USENIX Security (2025,
n = 576,000samples): arXiv:2406.10279 - Varun Magesh et al., Stanford RegLab & HAI, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools" (May 23, 2024): Stanford HAI Study
- Damien Charlotin, "AI Hallucination Cases Database" (accessed October 29, 2026): Charlotin Database
- The Guardian (reporting Moffatt v. Air Canada, 2024 BCCRT 149), "Air Canada must honor refund policy invented by airline's chatbot" (February 16, 2024): The Guardian Report
- RNZ, "Apple pulls AI-generated news notifications after generating fake headlines" (January 17, 2025): RNZ Coverage
- CBS News, "White House MAHA report cites studies that don't exist" (May 30, 2025): CBS News Report
- The Register, "Deloitte to partially refund Australian government over AI-assisted report" (October 6, 2025): The Register Coverage
- TechRepublic (reporting Oumi / New York Times SimpleQA AI Overviews study), "Google AI Overviews Inaccurate Answers Analysis" (April 21, 2026): TechRepublic Coverage
- Vectara, "Hughes Hallucination Evaluation Model (HHEM) Leaderboard" (updated May 2026): Vectara GitHub Leaderboard
