Put AI Completely In Charge? We’re Not Quite Ready For That

Can we, should we, leave at least some of our business processes and decisions to artificial intelligence, unattended by humans? Trust is important, and there is currently not enough of it in AI to manage most processes completely free of humans. This is unlikely to change anytime soon, experts agree.

Yes, we order things online, interacting with digital agents and processes all along the way. But we hope there is a human behind the scenes checking up on things. Likewise, we now drive cars with many autonomous features, but we hope the car’s functions are well-tested and monitored by their human creators.

Taking automation a step further, lights-out, hands-off processes driven by AI are a long way off, said Ed Ratner, head of machine learning at Verseon International Corporation. “Today, AI decisions are still neither transparent enough nor reliable enough for hands-off operation,” he explained. “As techniques become available that enable greater transparency and more sophisticated real-time monitoring of AI output, less human oversight will be required.”

But not all processes are equal. The urgency of human oversight can be delineated between low-stakes processes — in which processes could potentially be entirely AI driven — and high-stakes processes. “The degree to which that oversight is exerted may vary depending on the consequences associated with AI use,” said Joshua Wilson, Ph.D, associate professor at the University of Delaware. “For low-stakes applications, full automation or minimal oversight is appropriate. With high-stakes applications, the need for oversight increases.”

An example of a low-stakes application is automated writing evaluation in education, Wilson illustrated. “AI can provide valuable feedback to students without human oversight or involvement. Typically, feedback mechanisms are fully automated.”

However, Wilson continued, as the stakes increase, “the necessity for human oversight becomes more pronounced. In healthcare, for instance, while AI can assist in drug selection and dosage recommendations, a trained physician must monitor these decisions to avoid potentially lethal consequences. Similarly, AI-guided munitions should always be subject to human control to ensure ethical and safe deployment. In these situations, full automation may never be desirable, nor should it be.”

The challenge with minimizing human oversight is the fact that “many AI models are black boxed and developed without proper consideration for interpretability, ethics, or safety of outputs,” said Scott Zoldi, chief analytics officer of FICO. This elevates the need for responsible AI, that carry definitions of “what conditions some transactions have less human oversight and where others have more oversight.”

Even the highest-performing AI models “generate large number of false positives — mistakes — and so every output needs to be treated with care and strategies defined to validate, counter, and support the AI,” said Zoldi.

There are two complementary approaches that will help boost confidence in AI, said Ratner. “The first is having explainability of AI models. That means that users understand how the AI model is making decisions in terms of its inputs, both generally and in specific cases.”

The second approach, Ratner continued, “is constant monitoring of AI output. A number of new tools have come on the market recently that keep track of AI model output and can detect inconsistent or unusual outputs. The combination of explainability and real-time monitoring is an effective way to keep humans in the loop.”

Zoldi is aware of frequent instances when AI-driven decisions or processes were overruled or reversed by humans. “This happens all the time,” he said. “Responsible AI codifies all the essential human-driven decisions that will guide how AI will be developed. This includes approving or declining the use of data, removal of unethical relationships, and ensuring regulation standards are met.”

Part of this responsible AI process “also codifies details to a blockchain of how to monitor the AI in operation and the decision authority of human operators, which can include conditions where AI decisions are overruled and move to ‘Humble AI model.’”

Decisions about overruling or reversing AI typically should be left to people with appropriate domain experience, Zoldi stated. Organizations should also have “chief analytics or AI officers to set standards, help business units balance business risks and regulation, and enforce AI monitoring thresholds.”

“AI + human is the strongest solution, Zoldi said. “There should be no AI alone in decision making.”