Kerry Brown, Transformation Evangelist at Celonis, is a strategist and thought leader who helps companies achieve organizational excellence.
A new number has taken pride of place on enterprise dashboards: adoption. How many employees signed into an AI tool this week? How many prompts did they run? How many tokens did they consume? How many licenses are active?
For leaders under pressure to show their organizations are “doing AI,” these numbers can be reassuring, but they answer the wrong question. The real measure of AI success is whether AI makes the business work better.
I’ve spent my career on the people side of digital transformation, watching how jobs and organizations change when new technology arrives. My litmus test for any technology is simple: How do people’s jobs change? This translates into whether the technology helps someone do their work better, faster or smarter. Usage statistics may show adoption, but they don’t show whether the business is improving.
That distinction is the whole problem. A 2025 Boston Consulting Group study found that only 5% of surveyed companies were generating substantial value from AI at scale, while 60% were realizing hardly any material value despite significant investment. That’s not necessarily a verdict on the technology. Often, it reflects a more fundamental problem: The organization missed the mark on defining success before implementation began.
A more useful approach starts with three questions: What business outcome are we trying to change? How do we perform today? Did the outcome improve, and at what cost?
Activity Is Not Achievement
Metrics such as licenses, prompts and tokens are useful. They can indicate whether employees have access to AI, whether adoption is growing and where additional training may be needed, but they’re leading indicators at best.
The risk comes when organizations mistake activity for achievement. A rising prompt count doesn’t reveal whether customers are receiving better service, employees are spending less time correcting errors or an order is moving from placement to payment faster.
To understand whether AI is creating value, leaders need to measure what changes in the work itself. Depending on the process, that could mean cycle time, cost to serve, productivity, rework, customer satisfaction, cash flow or another clearly defined business outcome.
This requires a valid baseline: an honest picture of how the work performs before AI is introduced. Establishing one can be difficult because the way work is documented rarely reflects how it happens in practice. Real life includes exceptions, work-arounds, cross-functional handoffs and judgment calls that never appear on a flowchart or standard operating procedure.
Closing the gap starts with mining the digital records that enterprise systems already generate—ERP, CRM, ticketing, order management—to see how work moves from beginning to end, not how it’s expected to move on paper. With that operational view, leaders can compare performance before and after an AI investment and determine what improved, where work shifted and where complexity was added without value.
Building Measurement Into The AI Strategy
Leaders can make AI investments more measurable by asking three questions from the start:
1. What business outcome are we trying to change?
Begin with the business problem, not the AI capability. Identify the outcome that matters and define it precisely. “Increase AI adoption” isn’t a business outcome. Reducing the time required to resolve a customer issue, lowering invoice-processing costs or improving on-time delivery is.
Accountability needs to rest with a business owner. If no one owns the outcome, measuring progress and making decisions when results fall short becomes much harder.
2. How do we perform today?
Establish a baseline. Look beyond averages to understand where delays, rework and exceptions occur, as well as which parts of the process rely on human judgment.
This is a “facts are friendly” moment. The goal is to identify where technology can improve performance and where employees remain essential.
3. Did we get better, and at what cost?
Measure the same indicators after implementation, including the full cost of the change. An AI tool may make one task faster while creating additional review work downstream. It may create capacity in one team while moving a bottleneck into another, shifting instead of removing work.
An end-to-end view can help leaders see those trade-offs and also allow them to redirect resources toward use cases producing measurable results, improve initiatives that show potential and stop funding those that don’t.
From Experimentation To Intentional Investment
Testing and learning are part of innovation, and not every canceled experiment represents failure. However, without defined outcomes and a reliable baseline, leaders can’t distinguish a useful experiment from an initiative that’s simply consuming money, talent and attention.
Before celebrating the next adoption milestone, ask a more consequential question: What improved? Usage tells you whether AI’s being adopted. Outcomes tell you whether it deserves to be scaled.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?