AI Hallucinations: Why Language Models Make Things Up

In 2023 two New York lawyers filed a brief full of court decisions that had never happened. They had asked ChatGPT for precedents, the model obliged with names, docket numbers and quotes, and nobody checked them. The judge in Mata v. Avianca dismissed the case and fined the lawyers $5,000. It became the textbook example of an AI hallucination, but it was neither the first nor the last.

A year later a Canadian tribunal ordered Air Canada to pay a customer after its website chatbot invented a refund rule for bereavement fares. The airline argued the customer should have checked the real policy page. The tribunal disagreed: the chatbot was part of the website, and the company was responsible for what it said.

Three ways hallucinations cost money

The two cases above show the first two categories. The third one is less visible, and for developers it’s probably the most dangerous.

Invented sources. Citations, case law, scientific papers, URLs. They look perfectly formatted, because the model knows exactly what a citation looks like, it just doesn’t know whether this one exists.
Invented rules and facts. Refund policies, prices, opening hours, product features. A support bot that “fills in” a missing policy creates a promise the company may have to honour.
Invented code. Function parameters a library never had, command flags that don’t exist, and whole packages. A 2024 study generated 576,000 code samples with 16 models and found that at least 5.2% of package names suggested by commercial models and 21.7% by open-source models didn’t exist. Attackers noticed: registering a package under a commonly hallucinated name and waiting for someone to pip install it now has its own name, slopsquatting.

Why it happens

A language model has no internal database of facts to check against. It produces text by predicting the most likely continuation of what came before. When the right answer is well represented in the training data, the most likely continuation is usually also the correct one. When it isn’t, the model still produces something with the right shape, because producing text is all it can do.

That’s why hallucinations cluster around rare things: lesser-known people, exact numbers, specific dates, small libraries, local regulations. The model knows what a court citation or an npm package name should look like and generates one that fits.

Researchers from OpenAI and Georgia Tech add another explanation in Why Language Models Hallucinate. Most benchmarks reward correct answers and give nothing for “I don’t know”, so a model that guesses scores better than one that admits uncertainty. Training optimises for those scores, and the result behaves like a student on a multiple-choice test with no penalty for wrong answers.

Better models, sneakier errors

Each new generation of models hallucinates less, and models with web search or tool access can look things up instead of guessing. That’s real progress, but it doesn’t remove the problem, because the basic mechanism of generating text stays the same.

It also creates a trap. When a model was wrong every few answers, everyone double-checked. Once it’s right nine times out of ten, people stop checking, and the tenth answer goes straight into a court filing or a production deploy. The error rate went down, but the cost of each remaining error often went up.

What actually reduces the damage

The habits that help are not complicated, they just need to be applied consistently:

Ground the model in sources. Paste in the documentation, contract or policy and ask it to answer only from that text. Retrieval (RAG) and tools connected through MCP work on the same idea: look it up instead of remembering it.
Make “I don’t know” an acceptable answer. Say explicitly that admitting uncertainty is better than guessing, and ask where each fact came from. It isn’t a guarantee, but it shifts the model towards caution.
Verify the checkable parts. Every citation, every number, every URL. For code, run it, and check that every dependency actually exists on PyPI or npm before installing it.
Let agents test their own work. A coding agent that runs tests and reads the errors catches a large share of its own invented functions before a human sees them.
Don’t let a chatbot make promises. If a bot talks to customers about money, refunds or contracts, connect it to the real policy and make it hand off to a human when the policy doesn’t cover the question.

A property of the tool

It helps to stop waiting for hallucinations to be “fixed” and treat them as a known property of the tool, like rounding errors in floating-point maths. You use it anyway, you just know where it can bite you.

AI models are extremely useful for drafting, explaining, rewriting and writing code. The rule is simple: the more a mistake would cost, the more the final check has to be done by something other than the model itself, whether that’s a test suite, a source document or a human.