Zihan Wang is VP of Research & Partnerships at Abaka AI and Lead Director of the 2077AI Foundation, specializing in AI research and data.
The most expensive blind spot in enterprise AI right now is not the models. It is what we use to measure them.
Foundation models keep clearing higher capability benchmarks every quarter. The metrics that actually matter to a business deploying these systems are different ones: whether the model knows what it does not know, whether it stays accurate on your specific documents and whether it refuses when it should. On those, the numbers tell a harder story. Commercial LLMs hallucinate at rates between 15% and 52% depending on the task. EY’s 2025 Responsible AI survey found that 99% of organizations reported financial losses from AI-related risks, with an average cost of $4.4 million per company.
That is not a model problem. It is a measurement problem. And it is one every executive deploying AI now owns.
Why The Benchmark You Cite Is Probably Not Measuring What You Need
When an AI vendor sends you a deck, you will see scores on MMLU, GPQA or a handful of similar academic benchmarks. These were designed to test what a model knows, not whether it can be trusted with your data, your customers or your regulatory exposure.
I have seen what happens when a model that scores in the 90th percentile on a public benchmark gets dropped into a real workflow. The gap between what benchmarks measure and what businesses need is not theoretical, and it does not live where most leadership teams expect to find it.
Vectara’s updated leaderboard from late 2025 found that several leading reasoning models, including GPT-5, Claude Sonnet 4.5, Grok-4 and Gemini-3 Pro, had hallucination rates above 10% when summarizing source documents using a more challenging benchmark. The capability number went up. The trust number got worse.
If you are an operator making a buy decision today on the first number, you are taking on risk the vendor’s slide does not show.
What Scale Taught Me About Measurement
Years of building evaluation systems have taught me one lesson that holds up across every project: quality compounds faster than quantity. I have watched datasets grow past a million samples and seen the resulting model gain almost nothing in real-world reliability, because the examples at scale did not reflect the messy, ambiguous, domain-specific conditions where the model would actually be used.
The same dynamic applies to evaluations. A benchmark of clean textbook questions tells you almost nothing about how a model will behave on your contract review pipeline, your customer service queue or your medical coding workflow. Scale, in either direction, is only as honest as the cases you choose to test.
The companies that win this phase will not be the ones with the biggest training runs. They will be the ones with the most honest measurement.
Three Blind Spots Most Leadership Teams Have
If you are evaluating AI vendors right now, three questions almost no one asks tend to matter more than the headline benchmark.
First, what does the model do when it does not know? A 95% accuracy rate with confident hallucinations on the missing 5% is operationally worse than 85% accuracy with reliable refusals. Newer benchmarks like AA-Omniscience explicitly reward models for admitting their limits. From what I’ve seen, almost no commercial vendor cites them. They should.
Second, how does the model perform on your data, not the vendor’s data? Public benchmarks are training-data adjacent for most foundation models. The signal you need is a private, held-out evaluation set built from your own production traffic. If your vendor cannot show you performance on a domain test set you provide, you do not have a measurement. You have a marketing number.
Third, who built the evaluation, and what were they incentivized to find? An evaluation designed by the team that built the model will systematically miss the failure modes that team did not anticipate. Independent, adversarial evaluation is uncomfortable. It is also the only reliable signal of production readiness.
What A Better Evaluation Culture Looks Like
Treat evaluation as a continuous function, not a launch gate. Build a small in-house team whose job is to break your AI systems before customers do, with real production samples and domain context. Demand from your vendors not only leaderboard scores but their refusal behavior, their behavior under distribution shift and their performance on tasks designed to fail. Budget for measurement infrastructure the same way you budget for the models themselves. If you are spending six figures on inference and zero on evaluation, your AI strategy is a faith-based one.
This is the part of the AI stack the industry has underinvested in for years, and it is now the rate-limiting step for trustworthy enterprise deployment. The next breakthrough in AI for business will not come from a model release. It will come from the companies that finally start measuring what matters.
That work is unglamorous. It is also where I am betting the next decade gets decided.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?