Xin Guo is CEO of Solidigm.
Until recently, data center infrastructure was used mostly through direct human interaction: A person made a request, and the infrastructure handled it. That model has changed, with one recent report indicating that bot and agent HTML traffic is surpassing human traffic. That gap will continue to widen as trained models and agents are increasingly handling data, planning and execution.
AI is advancing to solve harder problems with increasingly advanced models, and those capabilities are improving quickly. But greater capabilities demand greater infrastructure, and without corresponding increases in compute, memory, storage and networking, those capabilities can’t be achieved. That is fundamentally shifting what we need from AI data infrastructure.
The Limits Of Legacy Infrastructure
We’ve seen many cases where a new model or capability is announced, only to discover that the existing infrastructure doesn’t support scaling. For example, this summer, one fast-growing AI provider had to pause new subscriptions just days after launching a model because demand pushed their GPU capacity to the limit. Single-threaded use of a new capability can look great, but when you try to deploy it across an organization, you have a problem, which can show up as slow or poor output.
A single prompt from one user in a chat application can generate a gigabyte or tens of gigabytes of context data cache. If you don’t have the infrastructure to store that information and provide it to the model promptly, your answers can be truncated. After a few turns, the system may forget what was already asked and answered. When the system has to re-compute answers to questions it’s already thought about, that’s wasteful and slows down actual throughput.
The same problem occurs when an AI system needs access to real-world enterprise data: If you don’t have enough capacity to store and serve that information, outputs will be less valuable. In the worst-case scenario, the model can start hallucinating.
The pressure is increasing as AI moves beyond text. A page of text might require only kilobytes, while a picture can easily reach tens of megabytes. Video multiplies that another ten-fold or even a thousand-fold, depending on resolution. Physical AI—robotics and real-world sensor data—amplifies data volumes even further.
A New Data Center Model
So what does the infrastructure need to look like? The physical footprint of the data center probably isn’t going to change dramatically, but the design thinking has to. In the old cloud era, most workloads were handled in memory, while storage was mostly for occasional long-term reference. Now we need to think about the entire AI infrastructure as a continuous pool of memory.
The difference between tiers is their proximity to processing power, particularly the GPU. The key challenge is serving data to the GPU at the right time and dynamically optimizing across tiers as a unified system. Think of it as a cascade from peak performance to maximum capacity. At the top is HBM memory, the hottest, highest-performance tier. Below that is a caching tier served by fast SSDs, followed by a tier served by high-capacity SSDs and finally archive storage. It’s a constant trade-off, and the challenge is optimizing those memory tiers to serve the entire workload.
Over the past year and a half, the bottlenecks have moved from compute to DRAM to networking; currently, high-capacity storage is one of the most critical choke points. Storage directly determines an AI data center’s economics. How much data you can store and actively serve to your compute determines your token throughput and output quality, and both determine your return on capital. If those metrics aren’t performing, you won’t be able to justify the application economics.
Building For Future Workloads
The biggest risk is locking in rigid infrastructure designs based on today’s workloads. The last thing you want is to shrink-wrap your design, locking in your AI interconnect based on the current understanding of workload demand. Instead, we need standardized solutions with flexibility and scalability built in, which lets us expand on demand across different data infrastructure tiers.
There is a lot of debate about whether this AI-driven demand and infrastructure growth is cyclical and will eventually slow down. Current data show that this is different from previous cycles, in that machine-generated traffic is already surpassing direct human interaction traffic across the web and in data centers. Demand is no longer limited by human population; it’s limited only by human imagination, which I would like to believe is unlimited.
There could be speed bumps along the way, including constraints around power or space. But the long-term trend is growing capability, and we’re really just at the beginning. That’s why advance planning for infrastructure is critical—today, not tomorrow.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?