Founder & CEO of Workmetrics, a leader in workforce software. Doctor of Information Technology specialising in data integration and AI.
Bringing AI into business environments was always going to yield a problem or two. Among the more discussed ones are LLM hallucinations, which are outright false facts generated when a model fills any gaps in its knowledge with assertions that sound plausible. But perhaps a less talked-about aspect of this digital plague of sorts is businesses locking their language models inside closed-loop systems.
Here, PDF archives, internal wikis, SharePoint repositories and all sorts of documentation reign supreme. And on paper, the setup is foolproof. If an AI is locked inside the corporate firewall, it can’t be tainted with external information. In reality, it creates a potentially dangerous illusion of security because there is no safety net anymore.
The majority of current workplace AI assistants share a few things in common, as opposed to the open internet. When data shows that hallucination rates range from 50% to 82% across models and prompting methods, can you afford to have your AI rely on ungoverned data sources?
You have to admit that the logic is comforting: Answers stay grounded in the company’s own approved material, which eliminates what is arguably best described as confident falsehoods. It’s a reasonable safeguard, since the assistant can’t make stuff up if it’s only reading a “homemade” employee handbook and technical specs.
A closed internal system doesn’t guarantee an error-proof system. The only guarantee is that the technology is entirely restricted to your universe, which brings us to the issue at hand: Is the company’s internal documentation that clean and up to date?
Hardly. In fact, there’s a case to be made that most knowledge bases are closer to digital versions of archaeological digs than contemporary policies, and a human can expose those gaps.
For example, as of July 1, 2026, Australia’s Paid Parental Leave scheme has expanded to 26 weeks (130 days), alongside increased weekly pay rates and higher income thresholds. But, an HR document may still list PPL entitlements as they stood a year or two ago.
A seasoned HR advisor catches the change and corrects it on the spot. That constant correction is the reason why “good enough” documentation truly felt good enough.
Unfortunately, AI has no such instinct. It doesn’t scratch its head and wonder if the document still makes sense. It simply repeats the figure it sees as fact, because within the information it’s allowed to use, that figure is the truth.
What causes AI to hallucinate?
At least in terms of internal documentation, training data is the prime culprit.
Anything past AI’s training cutoff is blank space, which the model graciously accepts as a personal invitation to improvise. This particularly holds true for hyper-specialized and underrepresented topics, where the model doesn’t admit defeat. It looks at adjacent data and launches a guess, a pitch-perfect wrong answer that oozes confidence.
Then there’s the way we talk to these systems.
A vague prompt gives the AI a lot of breathing room to drift away from reality. You have to remember: Language models are fundamentally optimized to sound coherent, not necessarily correct. Sure, prompt engineering helps by slapping guardrails on the scope or engaging in an AI version of cosplay by assigning it a specific “role.” Still, a well-engineered prompt aimed at an unmanaged stack of internal documents is nothing more than a roundabout path to a nicely written fabrication.
We humans are to blame as well.
When an internal tool answers a question in three seconds flat with neat bullet points and a polite tone, our brain automatically assumes it’s the real thing. Psychology has a name for this behavior: vigilance decrement. Human reviewers experience profound cognitive fatigue when a system is right the vast majority of the time. They switch off and start rubber-stamping the output that looks structured and polished.
Heck, the system said so, and with 100% fidelity while at it.
When does AI need to step back and a human step up?
With improvements in LLM processing (especially tokenization), hallucination rates will likely shrink. The question is: Will they drop to a percentage that is acceptable across the board?
Probably not. For the time being, the fix lies in a change of approach. Instead of treating enterprise AI like an all-knowing oracle, we should treat it like what it is: an incredibly fast but literal-minded digital intern who has read every company file but has zero intuition about what makes sense.
To get this right, though, there needs to be a clear line between automated efficiency and human judgment.
There’s no denying that AI excels at the heavy lifting, so it should stick to it. You know, things like high-volume data retrieval (nobody wants to sift through thousands of folders to find a specific vendor agreement or product specification) or the tedious and taxing work of translating staff questions into precise terminology across what are usually siloed systems.
Where it falls well short of the mark is reasoning across those silos. A human knows better than to mash together two metrics defined differently by two departments, which is what AI would do in a highly plausible fashion.
We need a smarter safety net.
The cost of weak documentation was always tangible; it’s just that it was paid by people until now. And before you draw the wrong conclusion, nobody is saying that humans are infallible. We miss things, too, sometimes often. But as it stands at the moment, one employee’s occasional error becomes AI’s instant and repeated error.
Real progress means understanding that AI is an amplifier of organizational reality. Its incredible power to weaponize internal documentation without a shred of malicious intent should give business leaders a hefty pause.
So maybe, just maybe, the smarter safety net doesn’t require lots of smarts. Maybe it’s accepting wholeheartedly that AI can accelerate how fast we find answers, and knowing when human critical thinking should take the wheel. It sounds like a system that respects its own limitations.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?