Does AI just make things up, and can we stop it?

Does AI just make things up, and can we ever stop it? That's the question this whole episode chases, and this first segment asks the hard version of it. When a model hallucinates, is that a bug we can engineer away, or is it baked into what these things are?

Start with the pessimists, because their case is a proof, not a vibe. In 2024, a team at the National University of Singapore published a paper with a blunt title: "Hallucination is Inevitable." Independent academic work, no company behind it. Their argument is mathematical. If you treat a language model as a general problem solver, one that's supposed to answer anything, then there will always be some inputs where it's wrong, because no computable model can match every possible truth. A second, separate group came at it from a completely different direction in 2025, using what's called the open-world assumption, and landed on the same conclusion. Two independent proofs, two different methods, one answer: you cannot drive hallucination to exactly zero. There is a floor, and the floor is above zero.

So that settles it, right? It's baked in. Except that's only half the story, and the other half is the interesting one.

Here's the question the proofs don't answer. That floor is real, but is it low or high? Are today's models hallucinating because they're bumping against a law of mathematics, or for some more embarrassing reason?

In 2025, a team at OpenAI published a paper called "Why Language Models Hallucinate," and it just cleared peer review in Nature in 2026. Now, flag this out loud: this is OpenAI explaining why its own product makes things up, and the conclusion they reach is, roughly, that we cause most of it ourselves through how we grade these models. A company has an interest in that answer, so keep that in your pocket. Still, the mechanism is peer-reviewed and it's sharp. Their claim is that we train these models like a multiple-choice test with no penalty for guessing. When a model says "I don't know," our benchmarks score that exactly the same as a wrong answer: zero. So the model learns the obvious lesson. If honesty and a confident bluff both score zero when you're unsure, you may as well bluff, because sometimes the bluff is right. We are, in their words, rewarding guessing.

Put the two findings together and the contradiction dissolves. The floor is inevitable, that part is proven by people with no stake in it. But today's hallucination rate is sitting way above that floor, and the reason it's up there is mostly a scoring incentive we built ourselves. Not a law of physics. A scoreboard.

So the honest grade on this one is contested, because credible people use opposite headline words, "inevitable" versus "fixable." But read carefully, they're not really fighting. One camp measured the floor. The other measured the gap between the floor and where we are now. The floor is inevitable. The gap is fixable.

Two caveats I owe you before I hand off. First, all those inevitability proofs are about a model answering from memory, alone. The moment a model can look something up or call a tool, it sidesteps the floor for that particular fact, so the theory is about the model as a closed oracle, which isn't how a lot of real systems run anymore. Second, the fix has a cost: a model tuned to say "I don't know" more often also refuses more often, and a model that won't answer isn't much use. Honesty and helpfulness pull against each other.

So what do you do with this? Stop expecting a future model with zero hallucinations; that's not coming, and now you know why. But also stop treating today's rate as fixed. A big chunk of it is a scoreboard problem, not a capability problem. Whitney takes the next question, which is the one that should really change how you feel about all this: when a model makes something up, how often does that actually happen? Because the honest answer is not one number.

Sources

• Xu, Jain, Kankanhalli, "Hallucination is Inevitable: An Innate Limitation of Large Language Models" (2024) [Independent], arXiv:2401.11817. Formal proof that a general-purpose LLM cannot be correct on all inputs; a floor above zero.
• Bowen Xu, "Hallucination is Inevitable" via the open-world assumption (2025) [Independent], arXiv:2510.05116. Independent route to the same conclusion.
• Kalai, Nachum, Vempala, Zhang, "Why Language Models Hallucinate" (OpenAI), arXiv:2509.04664, peer-reviewed in Nature 2026 [Vendor-funded, peer-reviewed]. Hallucinations are the expected result of training and evaluation that reward confident guessing over abstention. Grade: Contested (reconciled). Vendor framing: benefits a "solvable" narrative; flagged.
• Kalai & Vempala, "Calibrated Language Models Must Hallucinate" (STOC 2024) [Vendor-funded]. A calibrated model must hallucinate on facts seen once; abstention is the escape.

Michael just told you hallucination can't be zeroed but today's rate is mostly a scoring problem. So here's the number that scoring produces. How often does AI actually make things up? And I have to warn you: the honest answer is not one number, and the reason it isn't one number is the whole story.

Let me give you two rates for the exact same model, and watch how far apart they are.

First rate. You hand the model a document and say "summarize this, don't add anything that isn't in it." That's the grounded task, the model has the source right in front of it. There's an open, reproducible test for this called the HHEM leaderboard, run by a company called Vectara. Flag them out loud: Vectara sells tools that detect and reduce hallucination, so they have a business reason to talk about it. But the detector itself is open source and anyone can re-run it, so I'll trust the measurement and just keep their sales pitch in mind. On their current hard test, from May 2026, the best models hallucinate on roughly three to eight percent of summaries. A few do even better. So on the grounded task, this is close to a solved engineering problem.

Now the second rate. Same caliber of model, but this time you ask it an obscure fact from memory, nothing to look at. There's a benchmark for that from OpenAI called SimpleQA, and another peer-reviewed one from Meta called HalluLens, and both are vendor-built, so treat the exact digits as claims, but they've been reused independently across the field and the pattern is rock solid. When these models choose to answer a hard factual question from memory, they're wrong thirty to fifty percent of the time. Not three percent. Thirty to fifty.

Sit with that. Same model. Give it the source, it's wrong maybe five percent of the time. Take the source away, it's wrong thirty to fifty percent of the time. A tenfold difference, and the only thing that changed is whether you handed it the answer. Grounded is nearly solved. Working from memory is absolutely not. And when someone tells you "hallucination is basically fixed, it's down to one percent," they are quoting the grounded number and pretending it's the memory number. That is the single most common trick in the coverage.

Now the twist you won't expect. You'd assume the fancy new reasoning models, the ones that think before they answer, hallucinate less. They don't, necessarily. On that memory test, OpenAI's o3 hallucinated more than the older o1. And on the hard grounded test, a reasoning model called o3-pro sat at twenty-three percent, worse than models a tiny fraction of its size. More thinking did not buy more truth.

One more honest wrinkle, and then the verdict. A "hallucination rate" is partly a policy choice, not just a skill. Some models, Claude for instance, just decline to answer a lot of questions. That lowers their hallucination count, because you can't be wrong if you don't answer, but it also means they're covering less ground. So comparing raw rates between companies without asking "how often did it just refuse" is apples to oranges.

So the grade. The measurement is strong: we can measure this, the tests are reproducible, the numbers are real. But "is it getting better" is genuinely contested, because it depends entirely on the task, and the reassuring numbers mostly come from companies selling the fix. Grounded, it's improving and it's low. From memory, it's still bad and it is not uniformly getting better.

What do you do with that? Never trust a single hallucination percentage without asking one question: did the model have the source, or was it working from memory? Those are two completely different machines wearing the same name. Michael takes the last one, which is the practical payoff: given all this, what actually cuts hallucination, and what only feels like it does?

Sources

• Vectara Hallucination Leaderboard, HHEM-2.3, updated 2026-05-11 [Vendor: Vectara, open-source detector, reproducible]. github.com/vectara/hallucination-leaderboard. Grounded summarization: best models ~3 to 8%; reasoning models (o3-pro 23%) worse.
• Wei et al. (OpenAI), "Measuring Short-Form Factuality" (SimpleQA), 2024 [Vendor-benchmark], arXiv:2411.04368. Closed-book: 30 to 60%+ incorrect when answering; o3 above o1.
• Bang et al. (Meta FAIR/HKUST), "HalluLens: LLM Hallucination Benchmark" (ACL 2025) [Vendor, peer-reviewed], arXiv:2504.17550. Formalizes intrinsic (grounded) vs extrinsic (closed-book) hallucination; dynamic tests defeat contamination.
• Independent anchors: TruthfulQA (Lin et al. 2022) and FActScore (Min et al., EMNLP 2023) [Independent]. Grade: Strong (measurement) / Contested (trend, task-dependent).

Whitney just showed you there are two hallucination rates, a low one when the model has the source and a scary one when it's working from memory. So the practical question is this: what actually pushes those rates down, and what just feels like it should?

Start with the biggest real lever, and it's the obvious one. Give the model the source. Let it look things up and answer from what it found instead of from memory. This is the single most effective thing we know, and here's the honest catch the whole field agrees on: it reduces hallucination, it does not end it. There's an independent benchmark called RAGTruth, academic, no model vendor behind it. They handed models the correct documents and measured how often they still made things up. GPT-4, with the right source sitting right there, still fabricated in about nine percent of its answers. A tougher summarization test, this one from Vectara, so flag their commercial interest in the problem, put that number closer to forty percent. So handing the model the answer gets you a much better starting point. It does not get you a guarantee. You still have to check.

Second lever, and it's the sharpest idea in the field right now. Let the model say "I don't know," and stop grading it as if guessing were free. That's the Nature 2026 result Michael mentioned in the first segment. Flag it again: it's OpenAI, and it conveniently concludes the fix is to grade their models differently. But the logic is peer-reviewed and it's right. Change the scoring so honest uncertainty earns credit, and a whole category of hallucination disappears by construction. The idea is strong. The "and here's how much it dropped in the real world" evidence is still catching up.

There's also a detection tool worth knowing. An independent team at Oxford, published in Nature in 2024, found you can catch a lot of the confident nonsense by measuring the model's uncertainty at the level of meaning, not just words. It's genuinely independent work, and it's good, but notice what it does: it flags a bad answer after the fact, and only a subset of them. It's a smoke detector, not a fireproof house.

Now the myth. The thing that feels most like a fix is the one that falls apart. "Just have the model check its own work." Ask it, are you sure, double-check that. It feels responsible. A team at Google DeepMind tested it properly, and I want to flag that this cuts against their own interest, they'd have loved for it to work. It doesn't. A model reviewing its own answer with no outside information does not reliably get more accurate, and often gets worse. It flips right answers to wrong. The improvements people credit to self-correction almost always come from something outside the model quietly telling it when it's wrong. The model grading its own homework is not verification. It's the same guess in a mirror.

So here's the whole episode in one line. Hallucination can't be zeroed, that's proven. Today's rate is mostly a scoring problem we built, not a law of nature. And when you go to fix it, only one kind of thing actually works: verification that reaches outside the model, a retrieved source, a tool, a real check. Everything that stays inside the model's own head, including asking it to be more careful, is theater.

What should you do, concretely? Three things, sized to how much they help. If it matters, give the model the source and then check the answer against that source yourself, because even grounded it lies sometimes. Prefer models and settings that are allowed to say "I don't know" over ones that always sound sure. And never, ever trust a model's own second opinion as if it were a fact-check. The only opinion that counts is the one that came from outside the model. That's not a limitation you'll engineer away next year. That's the job.

Sources

• Niu et al., "RAGTruth: A Hallucination Corpus for Trustworthy RAG" (ACL 2024) [Independent], aclanthology.org/2024.acl-long.585. Grounded hallucination: GPT-4 ~9.3%, ~44% of responses hallucinated at least one span despite retrieved context.
• Bao et al., "FaithBench" (2024) [Vendor-benchmark: Vectara], arXiv:2410.13210. Summarization with source present: GPT-4o ~39% unwanted hallucination.
• Kalai, Nachum, Vempala, Zhang, "Why Language Models Hallucinate" / Nature 2026 [Vendor-funded: OpenAI, peer-reviewed]. Reward calibrated abstention; fix eval incentives.
• Farquhar, Kossen, Kuhn, Gal, "Detecting hallucinations using semantic entropy" (Nature 2024) [Independent, Oxford]. Detects confabulations (a subset) via meaning-level uncertainty; detection, not prevention.
• Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet" (ICLR 2024) [Vendor: Google DeepMind, against interest], arXiv:2310.01798. Intrinsic self-correction without external feedback does not help and often degrades. Grade: Strong.