Why Your AI Makes Things Up, and Why It Gets Worse

Hallucination is what happens when an AI does not know something and answers anyway. It doesn't flag the gap, it fills it, and it fills it in exactly the same confident tone it uses when it happens to be right. That's the whole thing. It isn't lying, because lying needs an intention to deceive and there isn't one in there.

Here is the version worth memorising.

A hallucination is a confident answer from a system that was never given a reason to say "I don't know."

That definition is narrow enough to argue with, which is the point. Plenty of people describe hallucination as a glitch, a rough edge that better models will sand off. I think that reading is wrong, and the rest of this explains why, because if you believe it's a glitch you'll wait for a fix, and if you understand the cause you'll change how you work today.

What hallucination is not

It's not the same as being out of date. A model that tells you the wrong price because its training data stopped last year made a stale claim, not an invented one. Both are wrong, but only one of them was fabricated on the spot.

It's not a search engine coming back empty. When Google has nothing, you get nothing, and the blank result is itself information. An AI chat has no blank result. There's always an answer, and its confidence tells you nothing about whether it's real.

It's not fixed by asking nicely. "Only use real sources" and "do not make anything up" are the two most common prompts in the world and they don't work, because the model has no separate internal flag for facts it holds versus text it generated. It cannot obey an instruction to consult a distinction it doesn't have.

And it's not rare. It's the normal behaviour of the system, running exactly as built. That's uncomfortable, but it's the honest starting point.

Why does an AI make things up instead of admitting it doesn't know?

Because guessing scores better than abstaining, and it was scored a very large number of times.

Think about how a model gets good. It's trained to continue text plausibly, and then it's measured on benchmarks that mark answers right or wrong. On almost all of those benchmarks, "I don't know" earns zero. A guess earns zero too if it's wrong, but it earns full marks if it happens to land. So across millions of scored examples, the strategy that wins is always to answer.

That's the argument in Why Language Models Hallucinate by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, later published in Nature. Their comparison is a student sitting a hard exam. If a wrong answer and a blank answer both score zero, you fill in something plausible on every question, every time. The model learnt the same lesson we all did.

Their proposed fix is not a smarter model. It's changing how the leading benchmarks are scored, so that saying "I'm not sure" is worth more than a confident wrong answer. That tells you how deep this sits. When the people who build these things say the fix is in the scoring, you can stop waiting for a patch.

So the mechanism, in four steps:

1. The model predicts text that looks like a good continuation of your question.
2. It has no internal marker separating things it knows from things it just produced.
3. Everything it produces comes out in the same confident register.
4. The training and scoring it went through rewarded answering over abstaining.

Nothing in that chain contains a step where the model checks whether it's right.

What happens when someone actually relies on it

Two cases are worth knowing, because they turn this from a curiosity into a business risk.

On 14 February 2024, the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada, 2024 BCCRT 149. Jake Moffatt's grandmother had died in Ontario, and that same day they went to Air Canada's website to book a flight from Vancouver to Toronto using the airline's bereavement rates. They asked the website's chatbot how it worked, and it told them:

If you need to travel immediately or have already travelled and would like to submit your ticket for a reduced bereavement rate, kindly do so within 90 days of the date your ticket was issued.

Air Canada's actual bereavement policy, sitting on a different page of the same website, said the opposite. It did not apply once travel was completed. Moffatt booked, flew, applied within the 90 days the chatbot had promised, and got nothing.

The airline's defence is the part everyone remembers. It argued it could not be held liable for what its own chatbot said. Tribunal member Christopher Rivers summarised the position bluntly: "In effect, Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission." He then pointed out that a chatbot is still just a part of the website, and it should be obvious that a company is responsible for the information on its own website.

He also dealt with the argument that the correct policy was findable elsewhere on the site. Air Canada, he wrote, never explained why one page should be more trustworthy than another, or why a customer should have to double-check one part of a website against another part of the same website.

The finding was negligent misrepresentation. Moffatt got C$650.88 in damages, C$36.14 in interest, and their tribunal fees back.

Small money, large principle. The company owns what its AI says.

The other one is Mata v. Avianca. In June 2023, Judge Kevin Castel of the Southern District of New York sanctioned two lawyers and their firm US$5,000 after they filed a brief built on six court decisions that did not exist. ChatGPT had produced them complete with case names, reporter volumes and page numbers. The detail that matters most for our purposes is that one of the lawyers asked the model whether the cases were real, and it confidently said yes, and pointed at Westlaw and LexisNexis as places to find them.

Asking an AI to check itself is asking the same process to run twice.

What is context rot, and why do long chats get worse?

Here's the second problem, and almost nobody warns you about it. Even when you've told your AI everything it needs at the start of a conversation, its performance drops as that conversation gets longer.

People call the everyday version context bloat. Researchers call it context rot, and there's real work behind the name. Chroma's context rot study, published in July 2025, tested 18 models across Anthropic, OpenAI, Google and Alibaba, from Claude Opus 4 and o3 down to smaller ones. Every model they tested performed worse as input length grew, on tasks that stayed exactly the same difficulty. Length alone did the damage.

Three findings from it are worth carrying around:

• One irrelevant distractor is enough to hurt. Not a wall of noise. One. And several compound.
• A well-organised long input is harder than a jumbled one. Models did better on shuffled, incoherent haystacks than on ones that flowed logically, which is the opposite of what everyone assumes.
• Focused prompts beat full prompts on conversational recall. Feeding back the whole history performed worse than feeding back only the relevant part.

Now put the two problems side by side. A long chat loses track of what you told it, and a model that has lost track still won't say so. It fills the gap. Context rot creates the hole and hallucination paints over it, which is why the wrong answers in hour three of a conversation feel so much more plausible than the ones in minute one.

The fix I actually run

I stopped keeping one long chat going. That's the whole first half of it, and it costs nothing.

The second half is that the things my AI needs to know about my business don't live in a conversation at all. They live in plain text files that get read at the start of a task. Delete every chat I have and none of that knowledge goes anywhere.

There are five, and none of them are clever:

1. The business file. What I sell, what I charge, who it's for, and what I've decided not to do. The last part earns its place, because it stops the same rejected idea being re-proposed to me monthly.
2. The voice file. How I write, with real examples of my own writing, plus the words and shapes I won't use. This is the one that changes output quality most and the one people skip.
3. The client files. One per client. History, context, what's been agreed, what they care about.
4. The decisions log. Dated entries recording what was decided and why, which is a different thing from a diary. It exists so I don't re-argue settled questions with myself.
5. The raw inbox. An append-only dump for anything worth keeping, tidied into the pages above every so often. Capturing in the moment and organising later are two different jobs, and trying to do both at once is why most people's notes die.

The mechanic is simpler than it sounds: the files are read at the start of a task, so the AI begins already knowing, instead of being told and then slowly forgetting. I wrote up how the whole thing is built, and how I use it day to day, in the second brain behind my AI agent.

If you want the short version of the habits, it's two lines. If the answer matters, make it show you the source. If the chat is long, start a new one.

When this does not apply, and what the files do not fix

Being honest about the limits is the part that makes the rest usable.

Context files do not stop hallucination. They narrow it. The AI still cannot tell you when it's guessing. What changes is that it now has your real numbers in front of it, so it has less reason to invent them. That's a reduction in the surface area, not a guarantee.

None of this matters for a lot of what you'll use AI for. Drafting, rewording, summarising something you pasted in, arguing with an idea, cleaning up your own messy paragraph. If the source material is in front of it and you're the one checking the output, hallucination has nowhere to enter. Most of my own daily use looks like this and I don't think about any of it.

It matters enormously the moment an answer leaves your desk unchecked. A price you quote, a policy you publish, a fact in a proposal, anything a customer or a court could hold you to. Air Canada's chatbot was fine right up until it wasn't.

And the files need maintaining. Mine get updated most weeks. A context file that's six months stale is a confident source of outdated answers, which is arguably worse than no file at all, because now the wrong answer has a citation.

For the business version of this: the thing to work out isn't whether your AI hallucinates, because it does. It's which of your outputs go straight to a customer without a human reading them first. That's a short list for most people, and it's the only list where any of this is urgent.

Everything else, you can just check as you go.