Why ChatGPT Makes Things Up, in Plain English, and What Actually Reduces It

On May 25, 2023, New York attorney Steven Schwartz filed a sworn affidavit in federal court explaining how six nonexistent airline-injury lawsuits ended up in his legal brief in Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. June 22, 2023) (CourtListener Docket). Schwartz had not simply trusted the first draft. He attached screenshots showing he asked ChatGPT, "Is varghese a real case," and "What is your source." ChatGPT replied that Varghese v. China Southern Airlines Co. Ltd. was a real Eleventh Circuit decision on Westlaw and LexisNexis. Two years later, on May 18, 2025, a syndicated summer insert inside the Chicago Sun-Times and Philadelphia Inquirer recommended 15 books; 10 out of 15 titles had never been written, pairing real novelists like Isabel Allende and Percival Everett with invented plots (NPR, May 20, 2025).

The gap between a model that guesses and one that abstains shows up clearly in primary benchmarks. In September 2025, researchers at OpenAI and Georgia Tech analyzed why frontier models still invent facts (OpenAI Research, September 5, 2025; arXiv:2509.04664). On OpenAI's SimpleQA benchmark (n = 4,326 short factual questions), an older reasoning model (o4-mini) said "I don't know" on 1% of queries and gave confidently wrong answers on 75% of queries. A newer configuration (gpt-5-thinking-mini) abstained on 52% of queries and cut confident errors to a 26% rate across the same 4,326 prompts.

As part of our practical guide to the two meanings of AI safety at work, this explainer skips linear algebra and vendor buzzwords. You leave with plain-English mechanics, empirical benchmark rules, worked time math, and the ed.cafe 4-Rule Grounded Prompt & Abstention Card.

How next-word prediction builds smooth sentences without a fact table

ChatGPT invents plausible falsehoods because a large language model stores word-pattern probabilities rather than a searchable table of verified records. When you ask a question, the software calculates which word fragment (called a token) most plausibly follows the words already on screen.

When payroll software looks up a start date, it checks cell D14. Either a date sits there or the cell is blank. ChatGPT has no cell D14. During pretraining, the system compresses billions of web pages, books, and code files into numeric weights representing how words cluster together. Common facts repeated thousands of times, like "The capital of France is Paris" or Excel SUM syntax, carve deep grooves into those weights.

Rare facts behave very differently. In "Why Language Models Hallucinate" (arXiv:2509.04664, September 2025), Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang proved that pretraining errors occur even when training text has zero human mistakes. A statistical generator estimates probabilities across millions of one-off details: local zoning fees, mid-tier papers, court dockets, or paperback titles. Kalai and colleagues showed in their study that the pretraining fabrication floor tracks the "singleton rate," the share of facts appearing once in the training sample. If 20% of biographical records in a training sample appear once, a closed-book generator forced to answer from memory will misstate at least 20% of comparable rare records.

Think of ChatGPT working from memory like a courtroom sketch artist drawing a witness three days after a trial. The artist knows human anatomy and suit lapels by heart, so every pencil line looks crisp. Where memory has a gap, like a tie stripe or a bailiff's four-digit badge number, the artist fills in realistic details so the drawing looks complete. Where that sketch-artist analogy breaks down matters at work: a human artist feels when memory turns into guesswork and warns you, "I guessed the badge number." A language model has no separate memory bank to check against its sketch. Every token comes from the same probability engine, whether repeating a capital city or splicing Isabel Allende's name onto an unwritten novel.

That mechanism explains why AI fabrications look like composite truths rather than gibberish:

• Real scholar + real journal + invented title: A real professor's surname and Journal of Applied Psychology sit close together statistically, so the model bridges them with a plausible paper title.
• Real court + real doctrine + invented docket: Six digits and a circuit label match the rhythm of real federal citations.
• Real code library + plausible method: If five Python libraries use .to_dataframe(), the model predicts .to_dataframe() for a sixth library that uses .export_table().

Why post-training exams train AI models to bluff instead of saying "I don't know"

Pretraining explains why model memory blurs on rare details, while post-training benchmarks explain why chatbots guess confidently instead of staying silent. Nearly all public AI leaderboards grade models like a multiple-choice exam that awards zero credit for leaving an answer blank.

Suppose a licensing exam awards +1 point for a right answer, 0 points for a wrong guess, and 0 points for writing "I do not know." Leaving Question 42 blank guarantees 0 points. Guessing gives a shot at +1 point with zero extra penalty. Over thousands of questions, bluffing every time wins.

Kalai, Nachum, Vempala, and Zhang audited major benchmarks including MMLU, GPQA, and SWE-bench (OpenAI Research, September 5, 2025). Almost all use strict binary 0/1 grading. If Model A declines to guess on 30% of obscure benchmark queries, binary scoring marks every abstention as 0, dropping Model A below a rival that bluffs on 100% of queries.

OpenAI demonstrated that exact split on SimpleQA (n = 4,326 fact-seeking questions from the GPT-5 System Card, arXiv:2509.04664):

On a conventional leaderboard, o4-mini edges ahead (24% correct rate versus 22% correct rate in 4,326 queries). At an office desk, o4-mini guesses on 99% of prompts in that test and invents wrong answers on 75% of queries. Meanwhile, gpt-5-thinking-mini declines on 52% of queries and cuts confident errors to a 26% rate. Even 26% in 4,326 closed-book queries remains far too high to send unchecked; see our guide on how often AI is wrong by task and sample size.

Why typing "be 100% accurate" or "are you sure?" fails in real tasks

Typing "be 100% accurate," "do not hallucinate," or "are you sure?" fails because stern commands neither supply missing facts into the chat window nor remove the pressure to produce a complete-looking answer.

Three empirical studies document why scolding prompts backfire:

1. Accuracy prompts inflate overgeneralization (n = 4,900 summaries): Uwe Peters and Benjamin Chin-Yee tested 10 language models in Royal Society Open Science (arXiv:2504.00025, April 2025). Models stripped hedges ("may," "in this cohort") and overgeneralized findings in 26% to 73% of summaries (odds ratio = 4.85 versus humans). Prompts explicitly demanding accuracy produced up to twice as many overgeneralized conclusions as plain prompts.
2. Asking "Are you sure?" in the same chat reinforces the error: When Steven Schwartz asked "Is varghese a real case" inside the same thread, his earlier fake citation stayed in the active context window (CourtListener Docket). Token prediction continues the active transcript coherently, inventing confirmations unless an external lookup intervenes.
3. Web search guesses behind paywalls (n = 1,600 queries): Columbia's Tow Center tested eight live AI search tools across 200 news excerpts (n = 1,600 queries, CJR Tow Center, March 6, 2025) and found wrong or unsupported citation answers on over 60% of queries. Similarly, an April 2026 Oumi audit of n = 4,326 SimpleQA searches on Google AI Overviews found a 91% accurate answer rate, yet 56% of those accurate answers cited pages that failed to back the claim (TechRepublic, April 21, 2026).

Even temperature = 0 with supplied text never hits zero errors. On Vectara's May 2026 HHEM-2.3 benchmark (n = 7,700+ long documents at temperature = 0, Vectara Leaderboard), error rates ranged from 1.8% to 24.2% of documents (7.0% to 13.6% across flagship models in that benchmark).

Four prompt and workflow shifts that measurably cut AI fabrications

Four workflow changes cut fabrication risk far better than scolding prompts because they supply primary text inside the window and flip the grading rule in favor of abstention.

First, paste the governing text directly into the prompt (after checking what not to paste into an AI tool at work). Open-book extraction moves you out of the 26% to 75% closed-book SimpleQA error band and into single-digit or low-teens grounded bands.

Second, change the grading rule so abstaining beats guessing: "If a fact, number, or condition is not stated in the pasted text, output [NOT IN SOURCE]. Outputting [NOT IN SOURCE] earns a 100% passing score on this prompt; guessing fails."

Third, require verbatim character-for-character quotes and section locators first so you can press Ctrl+F (Cmd+F) in your original file. Zero Ctrl+F hits immediately expose invention.

Fourth, split drafting from verification in a fresh window plus a human check using our pre-send check guide by output type.

Worked office math: 20 handbook FAQ answers

Here is how those rules change calendar time on a concrete office task:

• Assumptions: A clinic coordinator drafts 20 staff FAQ answers from an approved 38-page handbook (~14,000 words) inside a business AI workspace.
• Closed-book + "Be accurate" workflow: Drafting takes 6 minutes. Assuming a 30% closed-book error rate across 20 prompts (6 flawed answers) and zero page locators, cross-reading all 20 items against 38 pages takes 6 min x 20 = 120 minutes. Safe total: 126 minutes.
• Grounded workflow using the ed.cafe 4-Rule Card: Running the 4-Rule prompt on the uploaded handbook takes 8 minutes. On 3 silent topics, the model outputs [NOT IN SOURCE] (5 minutes to ask HR). On 17 items, Cmd+F quote and scope checks take 2 min x 17 = 34 minutes. Safe total: 47 minutes, saving 79 minutes. Practice this live in our 75-minute first hallucination lab exercise for adults.

Copy-paste card: The ed.cafe 4-Rule Grounded Prompt & Abstention Card

Pin the ed.cafe 4-Rule Grounded Prompt & Abstention Card beside ChatGPT, Claude, Copilot, or Gemini.

========================================================================
THE ED.CAFE 4-RULE GROUNDED PROMPT & ABSTENTION CARD
Free workplace safety artifact from ed.cafe (No Worker Left Behind)
========================================================================

DEFAULT AI EXAM SCORING VS YOUR DESK RULE:
- Default AI exam: Right = +1 | Wrong guess = 0 | "I don't know" = 0
  Incentive -> Bluffing always beats leaving a blank.
- Your desk rule:  Exact quote = PASS | "[NOT IN SOURCE]" = PASS | Guess = FAIL
  Incentive -> Honest abstention beats smooth invention.

THE 4 RULES BEFORE YOU PROMPT:
1. SUPPLY THE SOURCE: Paste approved primary text (never restricted PII/PHI
   into unapproved consumer accounts).
2. REWARD ABSTENTION: Require "[NOT IN SOURCE]" whenever the text is silent.
3. QUOTES BEFORE PROSE: Require exact Ctrl+F-checkable quotes and page headers.
4. COLD CHECK: Verify quotes in the primary file and a brand-new chat window.

------------------------------------------------------------------------
TEMPLATE A: OPEN-BOOK EXTRACTION & ABSTENTION PROMPT (COPY-PASTE)
------------------------------------------------------------------------
Task: Answer the questions below using ONLY the text inside <source>.

Grading Rules:
1. Do NOT use outside memory, web assumptions, or general industry norms.
2. If any fact, number, date, or rule is not stated explicitly in <source>,
   output exactly: [NOT IN SOURCE]. Outputting [NOT IN SOURCE] counts as a
   full passing score; guessing fails this task.
3. Preserve every qualifier ("may," "up to," "eligible hourly staff," dates).
4. Format each item with three fields:
   - Plain Answer (keeping all conditions)
   - Verbatim Quote (character-for-character from <source> for Ctrl+F check)
   - Section / Page Locator in <source>

Questions:
[1. Paste question 1]
[2. Paste question 2]

<source>
[Paste approved source text here]
</source>

------------------------------------------------------------------------
TEMPLATE B: FRESH-WINDOW FACT CHECK (OPEN A NEW CHAT SESSION)
------------------------------------------------------------------------
Task: Compare <draft> line-by-line against <source>.
List every number, date, quote, rule, and universal word ("all," "always,"
"never") in <draft>, and label each item:
- VERIFIED: Exact wording and scope match <source>.
- SCOPE INFLATED: Appears in <source>, but <draft> dropped a hedge or limit.
- UNSUPPORTED: Missing from <source> (remove before sending).

<draft>[Paste draft here]</draft>
<source>[Paste primary source here]</source>

------------------------------------------------------------------------
3 RED FLAGS THAT MEAN STOP AND OPEN THE PRIMARY FILE:
[ ] Ctrl+F / Cmd+F for a 5-word phrase from the "verbatim quote" finds 0 hits.
[ ] The draft cites a URL, statute, or study you have not opened yourself.
[ ] Cautious source words ("in this pilot") became universal ("all staff").
========================================================================

Where this stops being true

Grounded prompts cut everyday text fabrications sharply, yet four boundaries limit the ed.cafe 4-Rule Grounded Prompt & Abstention Card:

• Long pasted files still trigger single-digit to double-digit error rates: On Vectara's May 2026 HHEM-2.3 benchmark (n = 7,700+ long documents at temperature = 0), models still logged 1.8% to 24.2% hallucination rates (Vectara Leaderboard). Across 50+ pasted pages, models can merge clauses or miss middle-page exceptions. Always run Ctrl+F on extracted quotes.
• Data-privacy boundaries restrict what you can paste: Only paste internal files when they are public or inside an employer-approved business workspace with zero-training terms (plus a signed HIPAA BAA for patient data). Never paste SSNs, clinical charts, or secrets into consumer tools.
• Untrusted external files carry prompt-injection risk: Pasting external resumes, web scrapes, or vendor PDFs can expose the chat session to hidden instructions embedded inside the document.
• High-stakes domains require primary deterministic tools: This guide offers general operational information, not legal, clinical, or licensed engineering advice. Never rely on chat prompts for OSHA electrical/lockout math, clinical dosing, or unverified court filings. Adults building practical verification habits can view upcoming cohort details at our free Circle #1 starting page, where applications for 30 Massachusetts chairs close December 1, 2026.

Questions people ask

Key terms

• AI hallucination: A plausible-sounding statement, citation, number, or quote generated by a language model that is false or unsupported by primary records.
• Next-token prediction: The mechanism in chat models that generates text one word fragment at a time by calculating which fragment most plausibly follows prior words.
• Singleton rate: In Kalai et al. (2025), the share of specific facts appearing only once in pretraining data, setting a statistical floor for closed-book recall errors.
• Calibrated abstention: Training or prompting an AI model to decline answering (such as outputting [NOT IN SOURCE]) when evidence is absent rather than guessing.
• Grounded prompting: Supplying the authoritative reference text directly inside the prompt window and restricting answers strictly to that supplied text.
• Overgeneralization bias: The tendency of language models, observed in 26% to 73% of n = 4,900 scientific summaries, to drop qualifiers and turn narrow findings into broad rules.

Sources

1. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang, "Why Language Models Hallucinate" (September 2025): arXiv:2509.04664
2. OpenAI Research, "Why language models hallucinate" (September 5, 2025): OpenAI Index
3. U.S. District Court, Southern District of New York, Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (June 22, 2023): CourtListener Docket
4. NPR (Neda Ulaby), "Fake books on a newspaper's summer reading list were generated by AI" (May 20, 2025): NPR Report
5. Uwe Peters and Benjamin Chin-Yee, "Generalization Bias in Large Language Model Summarization of Scientific Research," Royal Society Open Science (March/April 2025): arXiv:2504.00025v1
6. Klaudia Jazwinska and Aisvarya Chandrasekar, "We Compared Eight AI Search Engines. They're All Bad at Citing News" (March 6, 2025): Columbia Journalism Review / Tow Center
7. TechRepublic (reporting Oumi / New York Times SimpleQA AI Overviews study), "Google AI Overviews Inaccurate Answers Analysis" (April 21, 2026): TechRepublic Coverage
8. Vectara, "Hughes Hallucination Evaluation Model (HHEM) Leaderboard" (updated May 2026): Vectara GitHub Leaderboard