For the past decade, the artificial intelligence landscape followed a remarkably simple recipe: if you wanted better AI, you built larger models, fed them more data, and added more GPUs. The formula worked, and it worked spectacularly.
Every new generation of hardware enabled larger neural networks, and larger neural networks broadened their reach. What began as image recognition evolved into agents capable of writing essays, generating software code, and creating realistic images and videos. The world grew accustomed to a simple equation: more GPUs meant more intelligence.
However, in the past couple of years, something changed. As we enter the fourth generation of AI—enterprise AI—the industry is discovering that the old recipe is no longer sufficient. The problem is not that GPUs have become obsolete. Far from it: GPUs will remain indispensable for training the giant, diverse models that underpin modern AI.
The problem is that enterprise AI asks these machines to solve a fundamentally different problem, and the assumptions that drove the generative AI revolution are starting to break down.
Enterprise AI operates by a different set of rules
Most people still think of AI as a chatbot: ask a question, receive an answer, end of interaction. That model defined the generative AI era. Enterprise AI, however, plays by different rules.
Imagine an AI assistant embedded inside a company. It starts the day by reading emails, analyzing spreadsheets, interpreting charts, reviewing contracts, listening to conference calls, and consulting internal databases. It collaborates with other AI agents, prepares reports for employees, and continuously updates its understanding as new information arrives. This is no longer a sequence of isolated interactions; it is continuous reasoning.
AI never truly stops working. It shifts constantly between tasks, maintains context over long stretches of time, and makes thousands of small decisions throughout the day. That may sound like a subtle difference.
But from a computing perspective, it changes everything.
Why GPU architectures struggle with enterprise AI inference
For years, AI systems were optimized for one metric: floating-point operational throughput, or FLOPS. The goal was straightforward: maximize the computation performed every second. GPUs excel at this because they were originally designed to process thousands of identical operations simultaneously.
Rendering video game graphics demanded enormous parallelism, and that same parallelism proved extraordinarily effective for training neural networks. Training, after all, is a highly parallel problem: huge volumes of data can be processed at once, and GPUs thrive under these conditions.
Inference is a different story. Inference is the stage where AI produces answers for users in real time. Unlike training, every new token generated by a large language model depends on the token that came before it. There is no shortcut around this sequential dependency. No matter how many GPUs are thrown at the problem, the system still generates one token at a time.
This is where a new bottleneck emerges. For decades, engineers focused on increasing computational performance while memory performance advanced at a far slower pace. Processors became extraordinarily fast; memory did not keep up. This imbalance is known as the memory wall.
Today, the memory wall stands as one of AI’s most stubborn obstacles. Large language models contain hundreds of billions of parameters, and some already contain trillions. Those parameters cannot fit inside a processor. Every time the model generates a response, it must continuously retrieve information from memory. Increasingly, AI systems spend more time moving data than performing calculations. In fact, in many deployments, far more energy is consumed transporting information than computing the answer.
This reality exposes a fundamental weakness of the GPU-centric approach. GPUs were built to maximize parallel computation; they were never designed to minimize data movement. During the generative AI era, the limitation could be partially hidden by batching many user requests together. Large batch sizes kept GPUs busy and delivered impressive throughput numbers.
Enterprise AI changes the economics
An AI assistant participating in a meeting cannot wait for hundreds of requests to accumulate before responding. A medical system helping a physician interpret an image cannot afford noticeable delays. An AI agent coordinating supply chains or financial operations cannot pause while batches form. Enterprise AI demands immediate responses. Latency becomes far more important than throughput.
Here lies the trap for GPU-based inference: to meet the latency constraints of the application, the GPU is forced to operate with small batches, and small batches impact the very utilization that made the GPU economical in the first place. The architecture is not broken; it’s simply being asked to run against its own design assumptions, and its performance degrades significantly as a result.
This shift is forcing the industry to ask a different question. Instead of asking how many operations a processor can execute per second, engineers are increasingly asking how quickly information can move through the system. It may sound like a small distinction. It is, in fact, a critical architectural change.
For years, AI hardware was designed around compute: the processor sat at the center of the system, and memory existed primarily to feed it. That philosophy is beginning to reverse. Fast memory access is becoming the center of architecture.
The goal is no longer to build faster arithmetic units. It is to increase data bandwidth and shrink the distance information must travel. Every millimeter matters because every movement consumes energy, introduces latency, and limits scalability.
The memory-centric revolution: From adding FLOPS to moving bytes faster
This realization is driving a remarkable wave of innovation across the semiconductor industry. Some companies are building processors with enormous amounts of on-chip memory to keep data local. Others are developing specialized inference engines optimized for autoregressive token generation.
New architectures activate only the portions of a model required for a given task, avoiding unnecessary computation altogether. At the same time, researchers are exploring near-memory and in-memory computing, two approaches that bring computation physically closer to where the data resides.
Among the many pioneers proposing new architectures, a few stand out. Cerebras has demonstrated that placing an entire processor on a single silicon wafer can dramatically reduce latency and communication bottlenecks. Groq has focused on highly deterministic, low-latency inference rather than peak throughput.
d-Matrix attacks the memory problem directly by bringing memory and computation closer together. SambaNova is exploring reconfigurable dataflow architectures, while Etched has aggressively specialized its hardware for transformer models.
Beyond GPUs: Rewriting the rules of AI inference
Another example is a next-generation inference architecture. It rests on three interconnected and mutually reinforcing innovations: a memory-centric design philosophy; a high-bandwidth, low-latency, deterministic pipelined data flow; and a hybrid compute fabric that fuses large arrays of tensor cores with on-the-fly reprogrammable DSP cores. Together, they address the fundamental bottlenecks that have long constrained the performance, efficiency, and scalability of AI inference hardware.
The first innovation, the memory-centric approach, marks a fundamental departure from conventional inference architectures. Traditional systems lean heavily on multi-level KV (key-value) caches shuttled through external memory, introducing latency and energy overhead each time the compute engines retrieve or update cached values.
The KV cache is not eliminated. Instead, its economics is transformed. Massive arrays of local registers, positioned in extreme physical proximity to their corresponding compute engines, allow the architecture to hold far larger contexts and model parameter sets close to the compute fabric, making every KV cache access dramatically more efficient.
This reduction in data travel distance cuts memory access latency, relieves pressure on the memory bus, and yields gains in power efficiency. The result is an architecture that sustains higher throughput while consuming less energy, an advantage in large-scale deployments where energy costs and thermal management dominate operational concerns.
The second innovation, the high-bandwidth, low-latency pipelined data flow, addresses the inefficiencies inherent in the single instruction, multiple threads (SIMT) execution model that dominates GPU-based inference today. SIMT architectures impose a heavy memory tax: threads must frequently synchronize and stall while waiting for memory operations to complete, burning both time and power.
By replacing SIMT with a deeply pipelined data flow, the chip ensures that data moves continuously and predictably through the compute stages, eliminating synchronization stalls and sharply reducing the latency and power penalties of traditional GPU memory access patterns. The pipelined approach also delivers deterministic, real-time inference performance, an increasingly important requirement as AI systems move into latency-sensitive applications such as enterprise AI, autonomous driving, robotics, and real-time speech.
The third innovation, the hybrid tensor core and DSP compute fabric, adds a new dimension of computational power and flexibility. Tensor cores are highly optimized for the dense matrix and vector operations that dominate transformer inference, but many workloads also require specialized signal processing, custom activation functions, and non-standard numerical transformations that tensor cores handle poorly.
By augmenting the tensor core arrays with reconfigurable DSP cores, reprogrammable on the fly and driven by a rich instruction set in which every instruction executes in a single clock cycle, the hardware can adapt dynamically to the computational demands of each model and each inference task.
This versatility carries weight for agentic AI. Pre- and post-processing tasks that would normally fall to the host CPU can be executed locally on the accelerator, minimizing CPU-accelerator traffic and freeing host compute capacity for other work. It also future-proofs the architecture against a model landscape that refuses to stand still.
Taken together, these three innovations from VSORA offer a purpose-built solution for modern, scalable AI inference in data centers and at the edge. The result is greater efficiency, lower latency, and broader adaptability than conventional GPU-centric approaches that translates into a lower cost structure and higher ROI for cloud service providers.
On the verge of next architectural transition
Computing history teaches that architectural transitions occur whenever a dominant technology encounters the limits of its original design assumptions. For decades, CPUs were the universal solution until graphics workloads grew so demanding that GPUs emerged as a new specialized architecture. Today, GPUs face a similar moment. They are not being replaced; they are being complemented.
The winners of the next AI race will not necessarily be the companies that build the largest clusters of GPUs. They will be the companies that build systems capable of the smartest data access and management, consuming less energy, responding faster, and reasoning more efficiently.
Lauro Rizzatti is a business development executive with VSORA, a technology company offering silicon semiconductor solutions that aim to redefine silicon performance. He is a noted chip design verification consultant and industry expert on hardware emulation.