Ido Susan, CEO and Co-founder, DriveNets.
The conversation about AI in boardrooms has shifted. Two years ago, executives asked what AI could do. Today, they’re asking how to optimize its cost. As AI continues to serve millions of customers, infrastructure economics, not model capability, will separate the companies that scale AI profitably from those that burn capital. However, from what I’ve seen in the industry, the way most organizations measure those economics is broken.
For decades, leaders have evaluated computing infrastructure through total cost of ownership benchmarked against raw capacity: How many processors are there? How many operations can they process per second? AI broke that model. Two clusters with identical GPU counts and near-identical spec sheets can differ dramatically in the useful output tokens they produce. That gap comes from how well the infrastructure matches the workload, from networking performance and memory usage to whether each chip is optimized for the job it’s assigned to do.
The metric that matters now is cost per token. The cost to produce 1 million tokens of output is the closest thing AI has to unit economics, and it’s the number executives overseeing AI investment are demanding from their teams.
Inference Changed The Equation
Why is this surfacing now? Because businesses have shifted from training models to running them. Training is a project; inference is forever. Inference runs continuously—on every request and for the lifetime of a product. At its annual Advancing AI event, AMD CEO Lisa Su noted that inference is now the primary AI workload, capturing about 60% of global compute. And according to Gartner, spending on inference will outpace spending on training in 2026. As AI continues to shift toward agents and always-on assistants, that share will only grow.
Inference has a contradiction built into it. Every request goes through two stages with opposite hardware appetites. The first stage, which digests the entire prompt, is bound by raw compute. The second, which generates the response piece by piece, is bound by memory bandwidth. A chip optimized for one stage is the wrong chip for the other. Run both on the same type of processor, as most clusters do today, and you pay for expensive silicon that sits partially idle on every single request.
One answer to this mismatch is heterogeneous AI, an architecture that uses accelerators from different sources or vendors in one central cluster. This architecture can then match each stage of the workload to the silicon best suited for it, rather than forcing everything onto one flagship chip.
This isn’t a fringe idea. Nvidia has paired its new Rubin GPUs, aimed at the compute-heavy first stage, with third-generation Groq processors purpose-built for low-latency response generation. I see this as the dominant GPU vendor’s own acknowledgment that one chip shouldn’t do both jobs. On top of this, Google split its eighth-generation TPU into two chips: one for training, one for inference. Amazon has offered separate training and inference silicon for years.
The performance case is just as clear. Peer-reviewed research from USENIX shows that splitting the two inference stages across separate GPU pools can serve up to 7.4 times more requests within the same latency targets.
What It Takes To Get There
Heterogeneous AI is easy to describe and hard to execute. There are three dominant challenges that I’ve identified in this space. First, there’s abstraction: Chips from different vendors behave differently, and a mixed fleet needs a software layer that hides those differences from everything above it. Second, there’s orchestration: The system must decide, request by request, which hardware should handle which stage, as demand shifts minute to minute. Third, we have networking: Splitting stages to different processors means handing off data between processors on every request fast enough that users never notice. That puts the network front and center, and it must be engineered as a first-class part of the architecture, rather than ordered last.
There is also a strategic dividend boards should not overlook. Once workloads can be placed across different silicon, you are no longer locked into a single vendor for your entire fleet. In a market where GPU supply and pricing are board-level risks, that flexibility is worth real money.
Where We’re Headed
My prediction is this: Within a few years, heterogeneous AI will simply be how AI infrastructure is built. A uniform cluster—one chip put everywhere and used for everything—will look the way the mainframe looks to a cloud architect today. Specialized silicon is becoming the norm, and the software behind it is maturing. Inference demand will only increase as AI agents become more prevalent.
The winners of AI’s next phase won’t be the companies with the most GPUs. Those that want to succeed need to reach the lowest cost per token and build the discipline to architect for those costs early. The capability race defined AI’s first act. The efficiency race will define its second.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?