A lawyer in New York filed a brief citing six court decisions. All six were fabricated by ChatGPT — invented case names, invented quotes, invented docket numbers. The model didn't malfunction. It did exactly what it was built to do: produce fluent, plausible text. The fabrication wasn't a bug in the system; it was the system working as designed, against a user who assumed fluency meant fact.
That gap — between sounding right and being right — is the whole problem. If you're putting generative AI in front of customers or inside a decision workflow, hallucination isn't an edge case you'll patch later. It's a property of the technology you have to engineer around from day one.
Stop calling it a bug
Large language models don't retrieve facts. They predict the next token given everything before it, optimizing for statistical plausibility, not truth. When the training data supports a confident, correct answer, you get one. When it doesn't, the model still produces a confident answer — it has no internal signal that says "I don't actually know this." Confidence and accuracy are decoupled.
This reframing matters because it changes what you build. You stop chasing a mythical "hallucination-free model" and start designing a system that assumes the model will occasionally invent things, then catches it before the invention reaches anyone who acts on it.
Match the guardrail to the stakes
Not every use case needs the same rigor. In my engagements, I sort applications onto a risk gradient and spend the budget where a wrong answer actually hurts.
- Low stakes — brainstorming, first drafts, code a human will review anyway. Light-touch. A confident wrong answer costs a few minutes.
- Medium stakes — customer support, internal knowledge search, document summarization. A wrong answer erodes trust or misinforms. Needs grounding and citations.
- High stakes — anything legal, medical, financial, or compliance-related, or anything that triggers an irreversible action. A wrong answer causes real harm. Needs grounding, verification, human sign-off, and an audit trail.
The mistake I see most often is applying uniform controls everywhere — either strangling a brainstorming tool with review gates, or letting a refund-approval agent run on raw model output. Calibrate to consequence.
The controls that actually move the needle
Ground the model in your data (RAG)
Retrieval-augmented generation is the single highest-leverage control. Instead of asking the model what it "knows," you retrieve relevant passages from your own trusted sources and instruct it to answer only from that context. Done well, this cuts fabrication sharply and gives you citations for free. Done badly — noisy chunks, weak retrieval, no instruction to abstain — it just launders bad context into confident prose. The retrieval quality is the ceiling on the answer quality.
Let the model say "I don't know"
Most base models are trained to be helpful, which pushes them toward always answering. You have to explicitly give the model permission — and instruction — to refuse. A system prompt that says "if the context doesn't contain the answer, say you don't have that information" is one line of text that prevents a whole class of fabrications. Then design your UI to treat "I don't know" as a success, not a failure.
Verify with a second pass
For higher stakes, don't trust a single generation. Run the output through a verification step: a second model call that checks each claim against the source, a rules engine that validates structured fields, or a factuality check on named entities and numbers. Yes, it costs latency and tokens. So does a lawsuit.
Keep a human in the loop where it counts
Human review isn't a failure of automation; it's a design choice. The trick is placing the human at the decision, not drowning them in every output. Surface the model's citations and confidence, flag low-grounding answers for review, and let people approve the actions that can't be undone.
Treat every generated statement as a claim to be verified, not a fact to be trusted. Build the system around that assumption and hallucination becomes a managed risk instead of a lurking liability.
Measure it, or you're guessing
You can't manage what you don't track. Before you ship, build an evaluation set of real questions with known-correct answers, and measure how often the system fabricates, how often it wrongly refuses, and how often it cites its sources correctly. After you ship, log outputs, sample them, and let users flag bad answers. A hallucination rate you're watching is a problem you can shrink. One you're ignoring is a headline waiting to happen.
Where to start
Pick your highest-traffic AI feature and answer three questions this week: What does a wrong answer cost here? Is the model grounded in trusted data, or improvising? Can it, and does it, say "I don't know"? If you can't answer all three, that's your first sprint. The goal was never a model that never errs — it's a system where its errors are caught, contained, and cheap.
Before your next AI feature ships, ask the uncomfortable question: what happens when it's confidently wrong — and who finds out first?