For most of the AI boom, the industry has talked about inference as if it were one problem, solved with one accelerator architecture. The problem was treated almost as a lookup table: what is the question, then look up the answer by traversing through the neural network once. Now, “disaggregated inference” is the industry buzzword.
But inference processing, with intelligent reasoning, tremendously long input sequences, and agentic AI workflows is not one problem. As AI matures the market is now learning, both painfully and profitably, that prompt ingestion and token generation are different workloads with different physics, different bottlenecks, and increasingly, different silicon. And the semiconductor industry will never be the same: We are entering a new age of silicon specialization. (Note that Nvidia and Cerebras, mentioned in this article, are both clients of Cambrian-AI Research, the author’s firm.)
That is why “disaggregated inference” is emerging as one of the most important architectural shifts in AI infrastructure. The idea is straightforward: run prefill, the compute-heavy prompt-processing stage, on one type of hardware, and run decode, the latency-sensitive and memory intensive token-generation stage, on another. The payoff is better utilization, lower cost per token, and faster response times for real users and agentic workflows.
Why the Disaggregated Inference Shift Is Happening
The economics of inference changed once AI moved from demo traffic to production traffic. In prefill, the model can exploit massive parallelism; in decode, it has to emit one token at a time, so memory bandwidth and latency dominate. A single generalized accelerator can do both, but it rarely does both as efficiently as bespoke hardware for each.
The realization of this drove Nvidia to invest $20B last December to acquire the assets and talent from startup Groq. SambaNova and Cerebras, two other memory-centric AI startups, began thinking in earnest how they could replicate this type of combination. SambaNova teamed up with Intel, and Cerebras is now joining forces with both AMD MI4xx GPUs and Amazon AWS Trainium. “If you want to play with the big boys,” Cerebras CEO and founder Andrew Feldman told me, “you gotta play with tremendous speed and memory bandwidth.”
Disaggregation gives operators a way to stop wasting expensive compute on the wrong phase. It also lets cloud providers and enterprises scale each stage independently, which matters when prompt lengths, user concurrency, and output lengths all vary by workload. In practical terms, that means better throughput, lower tail latency, and a more graceful way to serve large models at scale.
Two Primary Disaggregated Inference Hardware Camps
The first camp is **NVIDIA plus LPU. NVIDIA’s current direction is to keep GPUs as the versatile prefill engine while pairing them with an LPU-based decode layer orchestrated through Dynamo-style orchestration. The logic is clear: preserve the broad CUDA ecosystem, but hand the memory-bound generation problem to a specialist accelerator optimized for low-latency token output.
The second camp is AMD plus Cerebras, who is also teaming up with Amazon AWS for similar disaggregated inference using Trainium. Their newly announced AMD/Cerebras collaboration follows the same architectural logic but with AMD Helios for throughput and Cerebras wafer-scale systems for ultra-low-latency decode. The pitch is that high-throughput prefill and memory-optimized decode belong on different machines, and that the best inference stack is therefore a coordinated system rather than a single chip.
I would point out that since Cerebras is Ethernet-based, providing a plug and play interface, it may find other companies who need a disaggregation partner in order to compete.
Where Does Intel Fit In?
Intel is not leading the category, but it is trying to position itself as an enabler of disaggregated inference infrastructure. At Computex 2026, Intel highlighted rack-scale AI infrastructure built around Xeon processors and partner accelerators, and it showed a real disaggregated inference setup using Xeon for orchestration and execution, NVIDIA Blackwell for prefill, and SambaNova RDUs for decode. After all, you gotta play with the cards you were dealt.
That matters because Intel’s strongest role in this transition may be the control plane and systems layer rather than the headline accelerator. Inference disaggregation and agentic AI increases the value of orchestration, scheduling, networking, and CPU-side coordination, which plays directly into Intel’s enterprise footprint even if its own AI accelerator story remains less compelling than the leaders. Watch this space; Intel isn’t done yet.
And Where Does Google TPU Fit?
Google sits in a different place: it is both a hyperscaler and a silicon designer, so it does not need to sell a public inference story to the same extent as NVIDIA, AMD, or Cerebras. Its TPUs are already optimized around large-scale inference economics, and recent Google work on TPU inference has emphasized meaningful tokens-per-second gains from better inference techniques on TPU pods. But even Google may find itself compelled to reconsider, once we all see how well Nvidia and AMD do with their new partners.
Google’s structural advantage is that it can redesign the full stack around its own workloads, which makes it well suited to disaggregated serving inside its own cloud. The strategic question is not whether Google can benefit from this transition; it is whether it turns TPU efficiency into a broader commercial moat or keeps that advantage mostly captive to Google Cloud customers.
What Disaggregated Inference Means for Hyperscaler Silicon
Disaggregation strengthens the case for proprietary silicon, but it also raises the bar. Hyperscalers want chips that reduce cost at scale, free them from supply constraints, and let them tune the entire software stack to their workloads. Inference is now large enough, and valuable enough, that custom silicon economics look increasingly compelling.
At the same time, the workflow is fragmenting into specialized roles. If prefill and decode split cleanly, hyperscalers may decide they do not need one universal internal chip so much as a portfolio: general accelerators for broad compatibility, proprietary ASICs for scale economics, and software orchestration to stitch the pieces together. That creates an opening for a company like Google, which already owns the stack, but it also means the value is shifting from the chip itself to the system design around it.
The Strategic Takeaway
The big story is not that one vendor has won inference. It is that inference is becoming a systems business, and systems businesses reward specialization. AI training is a cost center; inference is the profit center.
NVIDIA is trying to keep control of the stack by pairing GPUs with LPUs; AMD and Cerebras are betting that throughput plus wafer-scale decode is the better answer; Intel is trying to own orchestration and enterprise infrastructure; and Google is using TPUs to protect its own economics inside a captive cloud.
For investors, the implication of specialization is that proprietary silicon is becoming less optional for hyperscalers, not more. But the real advantage will not come just from owning an ASIC. It will come from owning the disaggregated system, the compiler path, the scheduling layer, and the economics of serving the next token faster and cheaper than everyone else.
Disclosures: This article expresses the opinions of the author and is not to be taken as advice to purchase from or invest in the companies mentioned. My firm, Cambrian-AI Research, is fortunate to have had many semiconductor firms as our clients, including Baya Systems BrainChip, Cadence, Cerebras Systems, D-Matrix, Flex, Groq, IBM, Infleqtion, Intel, Micron, NVIDIA, Qualcomm, SImA.ai, Synopsys, Taalas, Tenstorrent, Ventana Microsystems, and scores of investors. For more information, please visit our website at https://cambrian-AI.com.
