By Dr. Rajesh Gharpure, Chief Delivery Officer at Persistent Systems.
Across enterprise teams, it is an ongoing debate whether an AI agent should be allowed to approve a loan, reroute a shipment, process invoices for a customer or resolve queries on HR policies. Few of those debates end with a clear answer. Most end with a temporary adjustment, keeping a human in the loop everywhere, for now, until someone works out the real rule. The challenge is that “the real rule” usually gets treated as a single policy question, settled once at the top.
It isn’t one. It’s five separate questions, asked repeatedly, one for each task an AI system touches. That lack of discipline has consequences. Deloitte’s 2026 State of AI in the Enterprise report finds that 74% of organizations expect to be using agentic AI within two years, but only 21% currently have a mature governance model for autonomous agents, one that defines clear boundaries for which decisions an agent can make on its own versus which require human approval.
Five Questions That Actually Decide It
Each one adds a layer, and none of them work in isolation.
• The first is criticality: How much does a wrong call affects regulation, finances, safety or customer perception?
• The second is predictability: What is a repeatable, rules-based task, or a judgment call with real ambiguity?
• The third is governance and accountability: Can every action be explained, and every decision undone if it’s wrong?
• The fourth is business maturity: Has the organization run automation long enough to govern something that acts on its own?
• The fifth, and the one leaders skip most often, is data quality: Do incomplete or fragmented data force more human oversight, no matter how good the model is?
Run a task through those five questions and it sorts into one of three lanes. When risk is low and predictability is high, the AI can execute on its own. When risk is medium and predictability is moderate, it can recommend and act, with a professional reviewing the exceptions rather than every case. When risk is high and predictability is low, AI should recommend and a human makes the final call.
That is the whole model. The discipline is applying it to every task instead of defaulting to keeping a person in the loop out of habit. McKinsey’s 2026 report finds that nearly two-thirds of respondents cite security and risk concerns as the top barrier to fully scaling agentic AI, well ahead of regulatory uncertainty or technical limitations.
Governance Built In, Not Bolted On
Here’s where most programs undermine themselves before they start. They try to add governance after the system is already live, as a patch when something goes wrong. It needs to be built into the design from day one, before the system ever goes live. For processes that touch financial data or anything confidential, that means the highest level of human interface from the outset. For something like customer support, a moderate level of human oversight is sufficient. Get that right at the design stage, and you cut rework later, not just risk.
This was evident in our work with a biotechnology firm automating dozens of processes across its core operations. Of the more than 50 use cases identified during process qualification, the team prioritized only those that could be automated with high accuracy, because the processes were sensitive and the cost of an incorrect AI decision was steep. For the most sensitive ones, where tolerance for error was lowest, humans were embedded by design at the major decision points. These checkpoints were placed exactly where a mistake would matter, rather than spread evenly across every step. Throughput rose by 30% with high error containment because the oversight was designed in rather than added on.
Readiness Has To Be Real, Not Assumed
Before any organization increases autonomy, it needs to be honest about readiness across three areas. On the technical side, is the data trusted and governed, and are security, observability and auditability actually in place, or assumed? On the operational side, are there clear escalation paths, and can a human quickly roll back an unexpected action? On the organizational side, is there clear ownership of AI-driven decisions, and does the workforce have the skills and expertise to effectively supervise these systems rather than simply approve their output?
To illustrate what genuine readiness looks like, consider a manufacturing firm we worked with. Its supply chain was diverse and operationally resilient, but decision-making was complex and entirely human-driven, which was the actual bottleneck. With fulfillment time as the firm’s North Star KPI, the team combined RPA, low-code tools and agentic AI to speed up inbound and outbound logistics, giving AI real autonomy over routing decisions. The one guardrail was simple. Any transaction above a set value required a person’s sign-off. One clear threshold did more work than an elaborate approval chain would have.
Business Process Owners Are The Custodians Now
As this plays out, the business unit owner’s role changes shape. The job now includes defining which decisions AI owns outright, which it shares with a professional and which stay under the full radar of a human. It also means managing a workforce of people and agents. Every AI decision has to stay explainable and auditable. Autonomy itself has to be watched, so it can expand or contract based on evidence. Process owners are becoming the custodians of enterprise autonomy, a responsibility that’s arriving faster than most teams expected.
Three things matter more than the rest. Establish clear decision rights before you delegate anything. Build governance and accountability into the system rather than around it. Scale through collaboration between people and agents rather than treating autonomy as a race to remove humans from the loop.
The enterprises that get this right have stopped asking, “Can AI make this decision?” They started asking, task by task, “Under what conditions should it?”
That’s the question that separates AI that scales from AI that stalls.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?