How to deploy AI safely despite hallucinations

Why your biggest AI risk isn’t the error, it’s the trust you place in it

There’s an assumption that creeps into nearly every boardroom the moment the words artificial intelligence come up: that a language model tells the truth. That assumption isn’t naive, it’s dangerous.

Large language models don’t occasionally make something up, they do it structurally, and with a confidence that has nothing to do with how much they actually know. Research shows this rate climbs to nearly fifty percent as a question becomes more open-ended, and that even trained professionals get swept along by it without noticing.

This signal shows what separates organizations that keep falling into the same trap from organizations where the application of AI actually holds up, backed by concrete figures and strategies you can apply today.

The problem isn’t that AI lies, it’s that nobody notices

The numbers are hard enough to be uncomfortable. Researchers at Renmin University introduced the HaluEval benchmark in 2023 and found that ChatGPT fabricated unverifiable information in roughly nineteen percent of the user questions examined [1]. A year later, with a larger dataset of nearly nine thousand questions spread across five domains, the picture turned out to be even more nuanced: in the biomedical domain the model hallucinated in about fifteen percent of cases, but in the open domain, where questions are less structured and broader in scope, that figure climbed to almost forty-seven percent [2]. That’s not an outlier you filter out with a few extra instructions. It’s a systematic pattern that shifts as the question opens up and the context thins out.

Anyone who dismisses this as a technical detail is missing the point. Whether an organization can trust an AI deployment with customer communication, medical advice, or financial reporting has nothing to do with how impressive a model sounds, and everything to do with how predictably it fails, and whether the organization is prepared for that.

Picture an experienced physician making a diagnosis with the help of a language model. The model is persuasive, it sounds authoritative, it cites clinical terms as if pulled straight from a textbook. Nothing in its tone gives away any uncertainty. That’s exactly the problem.

Automation bias: why trained people fall for it anyway

A randomized clinical trial published in NEJM AI put this scenario to the test with forty-two physicians, all of whom had received training in AI literacy. One group received correct recommendations from ChatGPT-4o, the other group was deliberately fed incorrect recommendations in half the cases. The result: diagnostic accuracy dropped from eighty-four point nine percent to seventy-three point three percent once the AI offered a wrong suggestion, a fourteen percentage point difference that was statistically significant [3]. These weren’t average users. These were physicians who knew AI can hallucinate, and they still fell for it.

That’s the core of what’s called automation bias: the more convincing a system sounds, the less critically the human next to it keeps checking. Human oversight gets held up as the ultimate safety net in nearly every governance discussion about responsible AI use, but in practice that net turns out to be far more porous than policy documents suggest. A reviewer who spends six hours checking off AI output develops the same blind spot as a pilot who leans too heavily on autopilot.

Why doesn’t this show up in the news more often? Because most hallucinations aren’t spectacular. They’re subtle: a wrong year, a fabricated citation, a summary of a study that’s just a touch too confident and misses the nuance. It’s exactly that kind of small, plausible error that’s most dangerous, because it never triggers the human’s instinct to double-check.

What the NEJM study actually shows is that “human oversight” as a concept isn’t enough unless that oversight is actively exercised. The physicians who failed didn’t lack knowledge of AI, in that moment they simply hadn’t run their own substantive check alongside the AI’s suggestion. A baseline level of subject-matter knowledge, or at least the habit of quickly verifying a striking claim yourself, for instance with a targeted search, remains indispensable. No prompt template or AI tool replaces that reflex, at best it reinforces it.

What actually works, and what only sounds good

This is where it gets interesting, because there’s a real difference between mitigation strategies that genuinely improve factuality and strategies that mostly sound reassuring on a slide.

Retrieval-augmented generation, having the model dig through an external knowledge source before it answers, is the most studied and most effective approach so far. Research on RAGTruth, a corpus of nearly eighteen thousand naturally generated RAG responses with word-level annotations, shows that even with retrieval, unsupported and contradictory claims keep surfacing, but that a relatively small model fine-tuned specifically on this kind of data can reach detection performance comparable to prompt-based approaches built on GPT-4 [4]. In other words: RAG doesn’t solve the problem, it shrinks it considerably, and how you combine it with targeted detection determines how much you actually gain.

What doesn’t hold up as a watertight solution is relying on the model itself to recognize its own errors. Research on semantic-aware cross-check consistency shows that hallucinations rooted in the question itself or in the model’s own limitations can’t be effectively caught just by checking whether the model gives the same answer consistently [5]. A model can give the same wrong answer three times in a row with just as much confidence as it would the right one. Consistency isn’t a synonym for truth, however tempting that equation is for anyone looking to quickly bolt on a guardrail.

The solution isn’t to distrust AI-generated output, but to govern its use with smart verification, human oversight, and provenance tracing.

And then there’s the category of solutions sold as a silver bullet that turn out not to be one in practice. Confidence scores, the percentages some interfaces display to indicate how sure a model is of itself, regularly correlate poorly with actual factual accuracy. Prompt engineering alone, however cleverly designed, changes nothing about the underlying mechanism that confuses probability with truth. And as for claims that ensemble voting, having multiple models vote on the best answer, offers a reliable solution for factuality in production environments: that evidence is currently thinner than is often suggested. There is strong evidence that voting across multiple reasoning paths improves accuracy on mathematical and logical tasks, but for open-domain factuality in production settings, independently replicated evidence with a clean before-and-after percentage is still missing. That’s not a reason to ignore the technique, but it is a reason to read claims about it with skepticism until that evidence exists.

Three prompt techniques that demonstrably work

Prompt engineering alone doesn’t fix the underlying mechanism, but that doesn’t mean every prompt has the same effect. There’s a difference between a prompt that simply sounds tidier and a prompt that forces the model into a verifiable process. The three templates below aren’t loose tricks, they’re derived directly from peer-reviewed research and are ready to copy and use.

The organizations already getting this right

Take a legal advisory firm that uses AI to scan draft contracts for deviating clauses. Instead of asking the model to generate a summary, which leaves room for improvisation and therefore for hallucination, the model is forced to tie every claim to a quote from the source document. No quote, no claim. That simple architectural choice, forcing the model into verifiable output instead of free generation, turns out in practice to be one of the most effective, low-effort interventions available.

Compare that to an organization that forwards AI-generated summaries to customers unfiltered because it’s faster and the first hundred examples looked fine. The difference doesn’t lie in the quality of the underlying model. It lies in the strategy behind the implementation: is there a structural boundary built in between what the model generates and what a human or a second system actually verifies, or does the entire chain lean on the assumption that things will probably be fine because they usually are.

A similar pattern shows up in customer service deployments. An insurer that deploys a chatbot to answer policy questions can choose to let the model phrase things freely based on what it believes it knows about the policy terms, or force the model to quote the current policy text verbatim and only then explain it in its own words. The first produces smoother, more natural-sounding answers. The second produces answers that feel slower but are verifiable, and therefore ultimately trigger fewer escalations to a complaints department. That choice is rarely made explicit in a project plan, but it’s what later determines which company ends up in the news with a painful example and which one doesn’t.

That assumption is exactly where things go wrong. A system that’s correct eighty-five percent of the time feels trustworthy, until the moment the remaining fifteen percent involves an invoice, a piece of medical advice, or a legal document that nobody double-checks anymore because it’s gone well so many times before.

What this means for anyone making decisions right now

The impact of this vulnerability reaches further than a single wrong answer. It touches on how much autonomy an organization hands to an AI system before a human sits in between who actually looks critically, not just on paper but in the reality of a busy workday.

Anyone deciding on AI deployment today can do a few things that don’t have to wait for more perfect models. Build verification in as an architectural design principle, not as a step bolted on afterward: a system required to back claims with traceable sources structurally outperforms a system that generates free text. Treat human review not as a checkbox but as a task that demands time and attention, because automation bias hits hardest the people who think they’re immune to it. And be skeptical of vendors who talk about guarantees against hallucination, because that guarantee simply doesn’t exist right now, no matter how convincing the marketing claim sounds.

Regulators are moving too, if slowly. The EU AI Act already requires some form of human oversight and documentation obligations for high-risk applications, precisely the two elements this research flags as most critical. But regulation by definition lags behind the pace at which these models are being deployed, which means the responsibility for now sits mostly with the organization doing the deploying.

What stands out about the organizations breaking this pattern is that they don’t treat hallucination as a bug that will eventually be fixed, but as a permanent trait they build structurally around. That’s a mental shift that pays off more than waiting for the next model update. The question is no longer whether a language model will occasionally make something up, that question has already been answered. The question that remains, and one every organization has to answer for itself, is whether the system built around it catches a fabrication before it does damage, or only gets discovered after the damage is already done.

The technology will keep improving, that’s a safe prediction given the pace of the past few years. But anyone waiting for hallucination to be fully solved before investing seriously in verification is waiting for something current research says isn’t in sight yet. The organizations winning right now aren’t the ones with the best model. They’re the ones that have built their systems so that it doesn’t matter which model runs underneath, because the truth always gets verified separately before it sees the light of day.

Related signals

• Why AI Models Fail Silently in Production Shows a related pattern: systems that appear to work fine until nobody checks why anymore.

• Who shapes the future of ethical AI? Places the trust problem from this signal within a broader framework of responsible AI deployment.

References

[1] Li J, Cheng X, Zhao WX, Nie JY, Wen JR. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. Available from: https://aclanthology.org/2023.emnlp-main.397/

[2] Li J, Chen J, Ren R, Cheng X, Zhao WX, Nie JY, Wen JR. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024. Available from: https://aclanthology.org/2024.acl-long.586/

[3] Automation Bias in Large Language Model-Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy: A Randomized Clinical Trial. NEJM AI. 2025. Available from: https://ai.nejm.org/doi/full/10.1056/AIoa2501001

[4] Niu C, Wu Y, Zhu J, Xu S, Shum K, Zhong R, Song J, Zhang T. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 2024. Available from: https://aclanthology.org/2024.acl-long.585/

[5] Zhang J, Li Z, Das K, Malin B, Kumar S. SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency. Findings of the Association for Computational Linguistics: EMNLP 2023. Available from: https://aclanthology.org/2023.findings-emnlp.1032/

[6] Dhuliawala S, Komeili M, Xu J, Raileanu R, Li X, Celikyilmaz A, Weston J. Chain-of-Verification Reduces Hallucination in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024. Available from: https://aclanthology.org/2024.findings-acl.212/