Shyam Ravindranathan is Senior Director of Product Management at SAP.
As companies deploy AI agents more, questions about human oversight have been growing. Specifically, I have seen the human-in-the-loop model built into the agentic AI development process in enterprise software become almost a rubber stamp as opposed to real, effective oversight.
Companies keep a person in the process for valid reasons, including concerns about jobs, credibility and hallucination. But the role is still not well defined. In many agentic AI systems, human oversight is treated as a checkbox or a liability-addressing measure, and reviewers are often overwhelmed and underqualified.
With the high volumes of output that AI can generate, reviewers frequently assume the AI is mostly correct, sample only a few items and batch-approve the rest. This can let the rare irreversible action slip through. And when accountability isn’t built into the heart of the system, it’s unclear who’s responsible if something fails: Is it the AI agent, the team that developed it, the person who reviewed it, or the business owner responsible for that entire set of products?
Right People, Right Decisions
It’s important that the intensity of oversight corresponds to the level of risk. Consider an AI automation step that approves a $400 purchase and another that approves a $4 million procurement order. If a person were handling both, you would naturally build multiple checks and balances around the bigger order. The same logic should apply to AI automation.
It’s also crucial to document every action an agent takes in a business process and assign a named owner for the result. Even though a step is automated, final responsibility for that action, even a small one, has to belong to a human being. Most customers I’ve seen start small and expand as they become more confident in their AI approach, so manual steps will remain.
Especially in the early stages, AI verification and evals must be done/created by true subject matter experts, just like any manufacturing process. You can’t expect just anyone from your development team or a generic cross-QA to perform an effective review. Controlling the volume of review processes is key even for experienced subject matter experts. No person can match an AI’s ability to correlate reams of data in a single step; An agent can look at the inventory, match it with what’s being produced and what’s in supply and flag a problem immediately. Ask a person to make that decision on the spot, hundreds of times a day, and they’ll start questioning their own capabilities and rubber-stamping.
Imagine an AI agent sees a supplier’s on-time delivery rate fall below a preset threshold and automatically places a backup order with the next-ranked supplier. What the data doesn’t show is that this second supplier is already near capacity from other manufacturers’ commitments. An experienced human buyer would have called the supplier first and learned they could cover only part of the order. An inexperienced reviewer approving hundreds of orders a day, however, is likely to wave the agent’s order through without that check.
Early in your AI journey, you shouldn’t compromise on the number of people assigned to human-in-the-loop (HITL) reviews. At that stage, it’s more important to have more people looking at smaller amounts of data, specifically when it’s risk-critical. Once the agent reaches some maturity, you can gradually reduce that number depending on effective evals and the issues arising from your AI agents.