Liran Zvibel, Cofounder & CEO, WEKA.
Latency has transformed from a performance metric into a measure of brand trust, and it can cost you customers.
There’s a trade-off hiding inside every AI deployment your team makes. Engineers know it intuitively. Executives feel the impact on their bottom lines. I’ve seen it stall road maps, bloat budgets and kill products before they ship.
It’s the “AI triad,” and it consists of three dimensions: latency, accuracy and cost.
For years, the operating assumption in enterprise AI was to pick two of these priorities. Want accurate outputs? Expect slower inference or higher compute spend. Want low latency? Sacrifice model depth or pay a premium for hardware to compensate. Want to control costs? Another side of the AI triad needs to yield right-of-way.
Here’s the problem. As AI-driven competition heats up in every industry, trading out latency is a non-starter. Inference is where latency multiplies, and that exposure is widening. Deloitte projects inference will account for roughly two-thirds of all compute this year, up from half last year. Similar to how a ChatGPT or Claude user won’t tolerate slow responses, your customers paying for AI products and services certainly won’t, either. Latency has transformed from a performance metric into a measure of brand trust, and it can cost you customers. It can also cost you millions in unplanned operating costs, sending AI budgets to stratospheric levels.
As a result, leaders are forced to give up either accuracy, which is critical as AI matures, especially in highly regulated fields such as finance and healthcare, or cost, an evergreen concern virtually all businesses work to limit.
Latency is the hidden tax creeping into every organization that is undermining innovation and throttling business results. But with the right approach in place, it’s a tax you don’t have to pay.
Why AI Trade-Offs Are Not A Law Of Physics
The AI triad is real, but trade-offs aren’t inevitable. They’re symptoms of infrastructure built for a different era when models were smaller, workloads were predictable and no one was running concurrent agentic workflows processing millions of tokens.
The core problem is distance: The gap between where data lives and where compute needs it creates a performance drag across every inference call. When context is “cold,” the model must process the entire prompt, including all interactions in the past, before generating output tokens. The key-value cache (KV cache) is essentially the model’s working memory: Keep it warm, and the model can reuse context. Lose it, and GPUs are forced to fully recompute work. Cold context means slow time-to-first-token (TTFT), leaving users waiting seconds for an answer they should have in under a second.
The business consequences follow directly. Every microsecond of latency slows your ability to generate insights, delays time to market and forces architecture teams to play it safe. Product managers start making hard calls about which features to cut, pushing innovation to the next release cycle and beyond.
What today’s AI demands is memory-class performance: data served at the speed of memory, not storage. Consistent microsecond response times. No recomputation from KV cache misses.
The Infrastructure Advantage Nobody Is Talking About
Most infrastructure conversations focus on peak latency, which is the best number you can hit under ideal conditions. This is the wrong metric. What matters for inference at scale is consistency. Maintaining microsecond latency across concurrent workloads, under real production pressure, is where the advantage compounds and cost control truly lives.
The numbers crystallize the problem. GPU high-bandwidth memory (HBM) operates at nanosecond latency. But it’s scarce, and once model weights and KV cache exhaust it, systems hit the AI memory wall, where cache spills past DRAM to storage. NVMe storage operates at 100- to 200-microsecond latency—a 1,000x gap. Traditional network storage is even slower.
When KV cache lives on the wrong side of this gap, every cache miss triggers an expensive and slow attention recomputation that can consume your latency budget. At thousands of concurrent requests, this isn’t a rounding error; it’s a CFO line item.
From here, the dominoes fall quickly. Memory-bound inference keeps GPUs busy but not productive, forcing over-provisioning of compute. Context that can’t be shared across requests means in-progress sessions restart cold, multiplying recomputation costs across your user base. In agentic workflows, one slow memory query easily becomes 50 queries, and cost per task balloons fast.
The way to break this cycle is to enable access to storage as if it were memory: Collapse the gap between data and memory so the system operates with consistent sub-100 microsecond latency. Once you achieve this, recomputation decreases, GPU productivity increases and over-provisioning vanishes. The associated tax drops precipitously.
Three Things You Can Do To Slash Latency Today
Follow these tips:
Auditing Your KV Cache Hit Rate Before Buying More GPUs
Before provisioning more GPUs, examine your KV cache hit rate. If it’s below 95%, you’re almost certainly paying a recomputation tax. This isn’t a capacity problem. More GPUs won’t fix a memory architecture issue and will simply scale the waste.
Tracking TTFT As A Product Imperative, Not An Infrastructure Metric
TTFT belongs on your product dashboard alongside conversion rates and churn. When engineering owns it in isolation, the business doesn’t feel the cost until it’s already embedded in customer expectations. Surface it in weekly product reviews and watch how quickly prioritization decisions sharpen.
Not Assuming Your Agentic AI Costs Will Look Anything Like Your Current AI Spend
Agentic workflows compound latency penalties in ways simpler AI tasks don’t. Model agentic deployments with realistic step counts and context window sizes. The cost picture will surprise you, and infrastructure decisions should reflect this reality.
Microsecond Latency: A Cost Strategy, Not A Performance Spec
When data stays close to compute, memory remains persistent, context stays warm and KV cache hit rates remain high, fundamentally improving the economics of AI inference. Latency drops without sacrificing accuracy. Cost per token falls as recomputation disappears. Longer agentic loops become not only viable, but efficient.
Fast, accurate and affordable AI isn’t a moonshot; it just requires overcoming the AI memory wall with smarter infrastructure decisions. The latency tax is optional. The only question left is how long you keep paying it.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
