Why Do LLMs Hallucinate?
Any of us who have played with the technology know the routine. “Hey, [insert chatbot], can I use [example] to do this?” And it shoots right back, “Absolutely! You just need to [example].”
So we try it. It doesn’t work. Maybe the thing it told us to use isn’t even real.
It’s a frustratingly common experience. We’ve been told these systems are “smart” (hell, it’s right there in the name) and capable. They have impressive-sounding names and inspiring classifications like “frontier models”.
People discuss quantization and comparative benchmarks with a fervor usually reserved for a huddled group of alchemists convinced they’re finally on the verge of turning lead into gold.
And then the thing confidently tells you that the software you’ve used for ten years has a menu option pointing to a function that simply does not exist. Never has.
Of course, more recently we’ve started seeing the pendulum swing the other way. Developers have spent enormous effort trying to train some of this behavior out, only to produce outputs that feel like gaslighting sycophants.
And yet it can talk to you. Just not always… correctly.
How can something seemingly so capable get something so basic so wrong?
First, what is an LLM actually doing?
Before we can answer that, we need to strip away some assumptions about what these models actually do.
They aren’t “thinking” in the way we normally mean the word, regardless of what the marketing department and design teams call the invisible process that’s burning your tokens before you get an answer.
They are predicting. That’s it. Just a big math problem… spitting out a prediction.
It receives tokens (your input broken into little digestible bits). These abstract pieces of language represented as math enter into an enormous network of learned relationships. That’s not as fancy as it sounds. It’s really just an unimaginably big set of numbers arranged in a many dimensional math equation.
To the model, it really doesn’t matter what those symbols refer to. In fact, to the model, each symbol likely doesn’t “mean” a thing in the way we imagine.
What it is measuring is more like the “shadow” of the ideas it was trained on. Tracing around them, outlining their shape from an incredible number of directions.
That’s what actually matters. And what matters to the work it does is the pattern of relationships, the shape of that “shadow of meaning” buried in language.
So something like “Paris” occurs in certain relationships with “France,” “capital,” “Europe,” “Seine,” “Eiffel Tower,” and millions of other linguistic contexts. Those relationships become encoded throughout the model during training and it can reliably predict “Paris is the capital of France” just as it can predict “The Eiffel Tower is in Paris.”
That is why any of this works at all, yes. But it also brings us directly to the problem.
Plausibility and factuality
Plausibility and factuality are related. Unfortunately for computer science they are not remotely the same thing.
Most of the time, that distinction is easy to miss because human language is constrained by reality. Our brains do a lot of work (without our realizing) to fill in those fuzzy little gaps and keep communication flowing and society (mostly) functioning.
But when we start measuring the shadow of the meaning buried in language, suddenly that fuzziness matters a lot more.
If millions of pieces of text contain relationships between France and Paris, those patterns provide a very strong signal.
But imagine asking for something much more obscure.
Maybe you ask for a specific feature in an old piece of software. Or a little-known court case. Or an academic paper on an extremely particular topic.
The model may have only been able to make a very weak measurement about the concepts you are actually asking for. But it may have some kind of information about what an answer to that question usually looks like.
And it almost certainly has an exceptional measurement of what an overly confident asshole on a random Internet forum sounds like. Or how a well-reviewed, authoritative academic paper sounds.
So the model can produce an answer that sits almost too well at the intersection of all of those patterns regardless of their compatibility in our minds.
That’s the crux of the problem. The model may have the shape of the answer much better than it has measured the answer itself.
And unless something else intervenes, generation continues anyway. Because the model’s only actual job (in the math equation that makes it up) is to make a complete response to the input it was given.
Why the problem is only going to get worse
This is also why hallucinations become more dangerous as models become better at language.
A bad language model producing nonsense is easy to distrust. If it wanders outside its core training material, it collapses completely and produces nonsense (legible or otherwise). We spot it immediately.
A very good language model can be wrong with impeccable grammar, appropriate jargon, clean formatting, and an explanation that fits everything you already understand about the subject.
For a non-expert, that can be almost impossible to detect. Sometimes it’s difficult even for an expert.
But while the model has access to more linguistic relationships than you or I ever will, you have access, directly or indirectly, to reality.
An LLM’s training data was not some pristine database of contextless objective truth. It was human communication: textbooks, documentation, conversations, articles, stories, arguments, mistakes, jokes, outdated information, competing interpretations and everything else we have managed to encode as language.
Information is not reality. Even our best information is just an approximation of it, and we’re always updating it.
And yes, we can move to the other side of the problem, once the model is trained.
We can connect them to search engines. Give them databases. Let them call calculators. Force citations. Make them route through retrieval systems.
But there’s still a problem there (maybe you’re seeing it). In this situation, the assumption is still that the model is the most powerful, most capable, most important part of the architecture.
The tools and databases are a thousand little fences to try and stem the tide of information (accurate or otherwise) that pours out of an LLM.
We’ll come back to that idea in a moment, but I want you to think about that assumption for a second.
Why is the model, the most unreliable piece of the puzzle, always put back at the center of the system?
So why don’t we just fix it?
“Well, just fix it by training the hallucinations away!”
This is where people jump to first. Which honestly makes sense on an intuitive level. You have an almost incomprehensibly complex information generation system. It can produce effectively infinite outputs.
“How can I possibly begin to control its output reliably?” is a reasonable concern.
So most people jump to the training data. After all, when humans make factual errors, the answer is to study and learn. To correct those errors.
Unfortunately, LLMs don’t learn like we do.
Let me show you what I mean. Suppose I tell a model:
Alice is taller than Bob. Bob is taller than Charlie.
Then ask:
Is Alice taller than Charlie?
I want it to generate something that may never have appeared explicitly in its training data.
That ability to move through learned relationships, to construct a useful answer instead of merely retrieving one, is a huge part of what makes an LLM useful. But that kind of problem is also exactly what LLMs have struggled with. Purely abstract linguistic reasoning.
That doesn’t mean they can’t do it, but getting reliable performance on what we think of as “simple” abstract reasoning has required enormous amounts of training and refinement.
The fact that it can do it at all is miraculous. Human language is cool.
And yes, we can improve the calibration. We can teach models to say “I don’t know” more often by including example conversations that end with people saying “I don’t know” (rare on the internet, I know). We can reduce bad training data, improve evaluation, constrain generation, and build systems that force verification when verification matters.
All of those things are useful! Absolutely.
But if your hypothetical solution to hallucination is that the model must never generate anything that cannot be directly recovered as an established fact, what you have described is no longer really the technology that made LLMs interesting in the first place.
You just described a database with an unusually expensive interface and a horrifically expensive and non-deterministic or even debuggable search process.
The deeper question
I’ll be honest. I really think we’ve been asking the wrong question.
Enormous amounts of energy have been devoted to understanding, isolating, measuring, and reducing the “hallucination problem” in a technical sense.
But as I mentioned the other week in my article comparing LLMs to the “screaming subconscious of humanity”, I think there is a dangerous assumption hidden underneath the entire conversation that’s getting skipped entirely.
When we ask, with such urgency, “Why do LLMs hallucinate?” there is another question hiding underneath it that I think is more important to answer first.
“Why did we assume an LLM should be capable of perfect factuality in the first place?”
I don’t think that conclusion naturally follows from what the technology is when we look at it honestly. We built a machine that can model the structures and relationships of human symbolic communication with extraordinary complexity.
Then we called it “artificial intelligence.” Because we’ve been feverishly imagining the concept, the implications, the possibilities, for decades now in R2-D2, Data, Robbie the Robot. Machines that could speak, because they could think.
So when we trained a model that can respond dynamically, conversationally, to language input… we almost immediately began judging it against an imagined machine that knows things.
But those aren’t the same thing. LLMs are a very real breakthrough, make no mistake.
They can do things with language and symbolic relationships that were effectively impossible with conventional software just a few years ago, yes.
But perhaps hallucination looks so “strange” and “frustrating” because we keep measuring these systems against the wrong imagined endpoint.
Maybe the answer is not to make the LLM solve the whole problem at all. Instead, give it the part it is unusually good at: turning fuzzy human language into clearer relationships.
So instead of asking:
“How do we make the AI stop hallucinating?”
we might get considerably further by asking:
“What kind of machine did we actually build, and what should we trust that machine to do?”
That is a much less magical question. But I suspect it is also a much more interesting one.