Why AI Success Still Appears So Elusive

Is artificial intelligence delivering on all its proponents’ promises? Business leaders, for one, say they are still intrigued by generative AI, but their enthusiasm is waning, a recent survey out of Deloitte suggests. Senior executives and board members who have “high” and “very high” interest in generative AI were 67% and 57%, dropping eight and six points from the first quarter of this year, respectively.

“This is likely due to many genAI efforts still being at the pilot or proof-of-concept stage,” said Jim Rowan, head of AI at Deloitte. In total, a majority of respondents said their organization has moved fewer than 30% or of their generative AI solutions into production at this time. “This makes it harder to see and market impact from these efforts,” Rowan said. “Until these efforts have matured and been scaled, it is a challenge for organizations to see if it is meeting their initial expectations.”

The bottom line is we’re still in the learning stages. “In the industries I serve, we’re all engaged in a massive, global experiment to see where generative AI and related technologies can create lasting business value,” Michael Umlauf, senior vice president of data science and analytics for TransUnion, told me. “As the experiments play out and we continue to learn more, including how to properly govern these tools and related systems, expectations will naturally start to drift downward and ground themselves more firmly in reality.”

Companies also keep learning more about the issues that tend to crop up around AI. “Enterprises are encountering all kinds of constraints, ranging from new governance requirements to shortages of skilled resources,” said Umlauf.

At the same time, Umlauf added, “we’ve seen a real willingness among our business partners and stakeholders to engage and learn from each other’s successes and failures.”

AI may be overhyped, “but it can be a game changer for those who figure out how it applies to their business,” said Courtney Machi, vice president at Andela, a global job placement network for software developers. “Those who have put in the resources to understand how to leverage it for their business are pulling ahead.”

Part of the challenge, Machi observed, “is many struggle with how to get started, the use case, and don’t have the right talent in their teams to tackle solving problems with AI. Others stop at internal productivity use cases using enterprise tech-like copilot, which are helpful, but difficult in terms of tying to ROI.”

Another factor that still has yet to be determined is measuring AI’s return on investment. “It depends on how realistic their expectations were to begin with, and whether or not they took the time to define clear objectives, performance metrics, and success criteria up front,” said Umlauf. “A lot of early successes have come in the form of productivity enhancements for knowledge workers, where benchmarking against existing practices should be a reasonably straightforward exercise.”

The current crop of generative AI tools “really show their versatility by helping coders write better code faster, writing initial drafts for content creators, and quickly extracting insights across sets of lengthy documents,” Umlauf added.

ROI is being seen within “companies who have been able to cut costs due to automation of previously human-driven tasks, or who have been able to grow top line due to differentiated product, and who have been able to solve for the fact that only 20% of AI output is accurate,” said Machi. “It goes back to whether or not companies are willing to invest time up front and willingness to take risk to some degree.”

Most organizations when adopting AI “have started with tactical benefits, such as improving existing processes and reducing costs,” said Rowan. “Essentially, they are hitting the low-hanging fruit to derive quick value while building knowledge, experience and confidence with AI. But now they are looking to scale to multiply that value, which comes with different challenges.”

Beyond straightforward productivity, more work is required to measure returns on more ambitious use cases, “to demonstrate lift versus incumbent solutions,” Umlauf said. Established solutions “have been honed to high levels of performance and accuracy over structured data sets, often over many years.”

As more AI projects “move beyond proof-of-concept and successful projects are deployed cross functionally, we will likely see additional impact and return on investment,” said Rowan. “Right now, executives may be making decisions based on a fear of missing out, meaning measurement will be a critical factor in maintaining interest and support from C-suite as the hype levels off.”

Realizing gains from AI involves “selecting the most promising use cases, diligently measuring AI’s performance versus alternatives, and committing to a process of continuous improvement,” Umlauf said. “In a narrow sense, enterprises can consider their AI projects successful when they are able to clearly and consistently demonstrate outperformance versus benchmarks and achieve wide adoption. Another sign of success will be when the AI recedes into the background and becomes yet another tool for enabling humans to achieve their goals more effectively and efficiently. Then we can look forward to the next wave.”