AI performance in real-world services is not determined by compute speed alone. Even the fastest GPU will sit idle if data cannot reach it quickly enough, and the gap between processor speed and data delivery is widening.
Research from UC Berkeley, ICSI, and LBNL, published in the paper "AI and Memory Wall," found that over the past 20 years, peak FLOPS in server hardware increased roughly threefold every two years. DRAM bandwidth grew only 1.6 times and interconnect bandwidth 1.4 times over the same period. Compute has outpaced the systems that feed it, and that imbalance is becoming the limiting factor in AI system performance.
Data moves through every layer of the system
In an AI system, data is never stationary. Training data, model parameters, user requests, and search results move between storage, system memory, accelerator-adjacent memory, and networks depending on the processing stage. When a user enters a question into a chatbot, the data center gathers previous conversation context, relevant documents, and search results before computation begins. The response speed users experience depends on how quickly that data can be collected and moved, not just on model quality.
The scale of these inference requests is substantial. OpenAI reported that 18 billion messages were exchanged on ChatGPT each week as of July 2025, and Google said Gemini surpassed 1 billion monthly users after August 2026.
Each layer in this flow has a distinct role. Storage holds large volumes of information. System memory provides a workspace for the server. Memory close to the accelerator feeds the processor rapidly. Networks connect multiple servers and accelerators into a single system. If any of these layers underperform, the entire system suffers - even with identical GPUs.
Bottlenecks appear across the stack
Insufficient memory bandwidth can leave an accelerator waiting for data, reducing utilization rates. Storage can slow response times when AI services need to retrieve large volumes of documents, images, or user histories. During large-scale training, checkpointing - periodically saving the model's intermediate state - can introduce latency across the entire training cluster if storage cannot keep up. MLCommons now measures storage performance separately through its MLPerf Storage benchmark, a sign that data management has become a core element of AI infrastructure.
Networks create similar constraints. In large-scale training, multiple servers exchange data simultaneously to divide a single task. Delays in one part of the system reduce overall processing speed. Power and cooling also constrain performance, since high-performance equipment requires stable power and effective thermal management to sustain peak operation.
System design determines real-world performance
The key to AI infrastructure is keeping accelerators continuously engaged in computation. When data arrives too slowly, accelerator utilization falls and performance drops relative to the investment. Data paths must be designed so that frequently used data sits close to processors, while large volumes of data are retrieved efficiently through storage and networks. It is impossible to place all information in the fastest location, so the placement and movement of data must be designed with speed, capacity, cost, and power efficiency in mind.
Software coordinates this architecture. Accelerator utilization varies depending on which tasks are processed first, when data is retrieved, and how work is distributed across devices. Inefficient scheduling creates new bottlenecks even with fast hardware. Intel notes that moving data within a system consumes energy without directly contributing to computation, which is why shorter data paths also improve power efficiency.
This shift is changing the competitive landscape of the AI infrastructure industry. Individual components - fast GPUs, high memory bandwidth, high-capacity storage, high-speed networks - are insufficient in isolation. Semiconductor, server, networking, cloud, and storage companies can no longer optimize only their own components independently. The required infrastructure architecture varies depending on which models customers use, what data they process, and what response speeds they need.
Memory companies face expanding roles. Beyond supplying products, they increasingly need to design data flows within customer systems and determine what memory architectures are required from a system-level perspective that includes processors, storage, and networks.
Why this matters for IT and development professionals
For professionals building and operating AI systems, the practical implication is clear: hardware selection alone will not determine performance. Capacity planning, data architecture, and system integration decisions now carry as much weight as GPU choice. Understanding how data moves through memory, storage, and networks - and how those layers interact - is becoming a core competency for AI infrastructure work. For those focused on the research side, the "AI and Memory Wall" findings offer a concrete framework for anticipating where system bottlenecks will emerge as models grow. Professionals in both areas can find relevant training through AI for IT & Development and AI for Science & Research.
The need for faster GPUs remains essential. But the question has evolved: how quickly and efficiently does data reach the GPU for processing? The next bottleneck determining AI performance begins with that question.
Your membership also unlocks:
