Su Belagodu, is the creator of the HITL Maturity Model and Humyn Pulse, she helps organizations adopt AI successfully.
The demo was flawless, the pilot numbers were strong, but six months into production, the same AI system was generating errors nobody caught because the human assigned to catch them had stopped looking.
I have audited more than 40 AI deployments across startups and enterprises, and this is the pattern I see most often. The technology rarely fails first. The oversight fails first, and the technology gets the blame.
The Adoption Paradox
MIT’s State of AI in Business 2025 report, published by the university’s Project NANDA, found that 95% of enterprise generative AI pilots delivered no measurable P&L impact, despite an estimated $30 billion to $40 billion in investment. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and the causes it names are escalating costs, unclear business value and inadequate risk controls. Model capability is absent from that list.
The industry’s standard explanations for these numbers are data readiness, integration debt and immature tooling. All of these things are real, but in the deployments I audit, one variable predicts post-launch degradation more reliably than any of them, and almost nobody tracks it: the quality of human oversight after go-live.
Note the word “quality.” Nearly every enterprise AI deployment now has a human in the loop. The question that determines whether that deployment survives is whether the human in the loop is actually functioning.
The Warm Body Problem
In a typical production system, a reviewer approves AI outputs at a pace no human can sustain. Override rates sit near zero, and leadership reads that as accuracy. What it usually means is exhaustion, trust that has drifted past the point of scrutiny or an interface where clicking “approve” is the path of least resistance.
This is “the warm body problem,” human presence without human judgment. It satisfies the compliance checkbox and nothing else.
The damage compounds, and errors that a fresh reviewer would have flagged in week one accumulate invisibly by month six, then surface as a customer complaint, a regulatory inquiry or a business decision built on hallucinated content that sailed through review. When the AI is wrong, and it will be wrong, the system’s real safety margin is whatever attention its human reviewer has left.
We Instrument The Machine And Ignore The Human
Every AI deployment review I attend has dashboards for model accuracy, latency and cost per inference. I have yet to walk into one with a dashboard for oversight health: override rates by design versus by accident, reviewer fatigue, or whether human corrections feed system improvement or die in a log file.
We built an entire measurement culture around the machine and left the human half of the system uninstrumented. That is the blind spot, and regulators found it before most boards did. Article 14 of the EU AI Act requires oversight that is effective, not merely present; rubber-stamp review does not qualify. The NIST AI Risk Management Framework treats oversight the same way, as something to be measured and managed rather than declared.
Organizations approaching this as a documentation exercise are certifying oversight structures that do not function, and the enforcement era has already begun.
Agents Raise The Stakes
Everything above describes AI that produces outputs for a human to review. Agentic AI takes actions, including sending messages, modifying records and executing multistep plans. The oversight burden does not grow linearly with agents. It compounds.
When agents run in parallel, human attention becomes the bottleneck rather than compute. When an agent misreads a goal and executes a flawed plan faithfully across 20 steps, catching the bad output at step 20 is too late; the misalignment had to be caught before execution began. And familiarity works against you; the longer people work with an agent, the more readily they approve its actions, whether or not its reliability has actually improved.
Companies that never built functioning oversight for output-generating AI are now granting autonomy to action-taking AI. Forbes revisited Gartner’s cancellation forecast earlier this month and reached a similar conclusion: The projects that survive will be the ones with a name on the override switch.
Based on what I see inside these systems, I would call the 40% estimate conservative.
The Turnaround Is Measurable
The fix is not more humans, and it is not more approval gates. Pile on checkpoints and reviewers start auto-approving, a failure mode I see so often I gave it a name: gate fatigue. More friction placed badly is worse than less friction placed well.
The fix is treating human oversight as a production system with a maturity path and a health score. In my framework, organizations move through five levels, from “automation-first,” where oversight exists on paper only, to “adaptive governance,” where the system monitors whether its own oversight is degrading. The most consistent finding across every audit I have run is the gap between where leaders believe they sit on that path and where their override data says they sit.
Closing that gap starts with three moves: Make oversight quality a KPI by tracking override rate, escalation accuracy and reviewer capacity alongside model performance. Budget human attention as the scarce resource it is by capping review volume and monitoring reviewer fatigue. Finally, route only genuinely uncertain decisions to people.
The Question That Separates The 5%
Ask any AI leader for their model’s accuracy and they will quote it to the decimal. Ask for their oversight score and the room goes quiet.
I have asked that question to more than 40 organizations. The handful who could answer it are, without exception, the ones whose deployments made it past the pilot graveyard.
Boards now ask every AI vendor, “How accurate is it?” The organizations that turn their AI investments around in the next two years will be the ones that learned to ask the second question, “How healthy is the human oversight around it?”
One of those questions has an industry built to answer it. The other is still wide open.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?