Swapnil Upganlawar is a senior manager in generative AI at Amazon Web Services (AWS).
Two experiences have shaped how I think about evaluating AI systems, especially in understanding how much responsibility to give an AI agent.
The first was when, in a previous role, I used MATLAB segmentation in location proteomics research to help classify proteins by location. While the results initially looked promising, human review led to a different assessment.
Later, at a consulting firm, we built a recommendation engine that improved the member experience. However, human review revealed that some parameters were overfitted, and an explicit feedback loop was critical. The system could be useful, but it still needed corrections that its initial results had not made obvious.
Before deploying agentic AI systems, it’s crucial to understand how the team will find mistakes after deployment and whether that feedback can change what the system does.
Structuring Human Approval
This type of oversight becomes even more consequential when an agent can change a record or move money.
Consider an agent handling customer refunds. Drafting a reply to a customer gives a person something to review, but actually issuing the refund changes a customer’s account.
Before allowing that second action, you need to define the cases the agent may handle and the evidence it must have. Missing order records or an ambiguous policy should send the case to someone who can resolve it.
The approval needs more evidence than a high completion rate. A 2026 study of AI agent reliability, covering 15 models and two benchmarks, found only modest reliability gains alongside improvements in capability. Those results are specific to the tests, but they support a practical concern: A more capable agent can still fail in ways that make additional authority hard to justify.
For the refund example, a better metric might be completed transactions. Did the correct customer receive the right amount? Was any required approval obtained? The original tau-bench research, for example, checks the resulting database state against the intended outcome and tests consistency across repeated attempts.
Finally, for a business process, that review needs to cover the authorization record, too. A correct payment can still have bypassed a required control.
Understanding What Needs Human Attention
My experiences shared above also make me wary of reviewing only the cases a system flags for help. That leaves the system deciding which parts of its own work get scrutiny.
Instead, I suggest sampling apparently successful cases as well as escalations, then use the findings to revise the rules for what the agent may do. Otherwise, mistakes can remain invisible in a report that looks reassuring.
Cost deserves the same attention. A 2024 research paper, “AI Agents That Matter,” found that most agentic evaluation practices focus too narrowly on raw accuracy while evaluating accuracy and cost should be evaluated together. For a deployment decision, the calculation should also include the time people spend checking, correcting and completing the agent’s work. That time can be easy to miss if the engineering budget covers the software and another team absorbs the exceptions.
Take a hypothetical batch of 1,000 refund cases in which the agent produces 850 verified outcomes without human correction. People resolve another 100, and 50 remain unresolved. If software and infrastructure cost $400 and validation, review and recovery cost $1,600, the combined process delivers 950 verified outcomes for $2,000, or about $2.11 each. These illustrative figures exclude initial implementation and later consequences of undetected errors.
While reporting 95% completion in this case would describe the combined process accurately, it would leave out the human contribution unless the report showed it separately. Even the 850 cases that required no correction might have needed approval.
Those distinctions need to be visible before the result can support a larger deployment.
Restricting the agent to routine refunds can be a good decision, provided the comparison includes the harder work left with people. Extending its authority to disputed transactions requires another evaluation. The economics of easy cases cannot tell us how much time the difficult ones will consume.
Finally, someone needs to own the review queue. Reviewers need enough evidence to understand what happened and a way to finish the case without repeating all the work. Their capacity belongs in the deployment plan. If cases wait for approval or correction, customers still wait, however quickly the agent produced its answer.
Conclusion
Based on my experiences with agentic AI, my preference is to begin with proposed actions, move to human approval and allow execution within defined limits once the results justify it.
Some cases may continue to require approval. A change in policy, a growing backlog or newly discovered errors should trigger a review of permissions already granted.
Before expanding agentic authority, ask the business owner responsible for the process to walk through the unresolved cases and the mistakes found during review. What changed because of that feedback? The next decision about autonomy should depend on whether those problems have been addressed and whether the people handling exceptions can support the additional work.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?