Processors

Inference 2.0: How enterprise AI is reshaping AI system architectures

by

Predictive, generative, and agentic AI are not successive generations. They coexist inside enterprise systems, each placing fundamentally different demands on the underlying compute infrastructure. The enterprise context layer A consumer chatbot relies primarily on pretraining weight memory and optional web search. An enterprise system must operate against information absent from pretraining: engineering specifications, financial ledgers, … Read more

Enterprise AI: Reshaping AI processor architectures

by

For the past decade, the artificial intelligence landscape followed a remarkably simple recipe: if you wanted better AI, you built larger models, fed them more data, and added more GPUs. The formula worked, and it worked spectacularly. Every new generation of hardware enabled larger neural networks, and larger neural networks broadened their reach. What began … Read more

MLPerf and the rise of latency-aware LLM benchmarking

by

Any discussion of modern AI system performance must include MLCommons and its MLPerf benchmark suite, which has become the industry’s de facto standard for measuring machine learning performance. Since its debut in 2018, MLPerf has provided a neutral, peer-reviewed framework for comparing hardware and software platforms across a broad range of AI workloads. The original … Read more

GPUs: A high-throughput architecture confronting a workload shift

by

There is a growing architectural tension at the heart of modern AI infrastructure. The processors that enabled the deep learning revolution—graphics processing units (GPUs)—remain the dominant engines of large-scale training and inference. Yet the computational profile of frontier language models is evolving in ways that increasingly expose the structural assumptions embedded in GPU design. Memory … Read more

The truth about AI inference costs: Why cost-per-token isn’t what it seems

by

The AI industry has converged on a deceptively simple metric: cost per token. It’s easy to understand, easy to compare, and easy to market. Every new system promises to drive it lower. Charts show steady declines, sometimes dramatic ones, reinforcing the impression that AI inference is rapidly becoming cheaper and more efficient. But simplicity, in … Read more

Round pegs, square holes: Why GPGPUs are an architectural mismatch for modern LLMs

by

The saying “round pegs do not fit square holes” persists because it captures a deep engineering reality: inefficiency most often arises not from flawed components, but from misalignment between a system’s assumptions and the problem it is asked to solve. A square hole is not poorly made; it’s simply optimized for square pegs. Modern large … Read more

The role of AI processor architecture in power consumption efficiency

by

From 2005 to 2017—the pre-AI era—the electricity flowing into U.S. data centers remained remarkably stable. This was true despite the explosive demand for cloud-based services. Social networks such as Facebook, Netflix, real-time collaboration tools, online commerce, and the mobile-app ecosystem all grew at unprecedented rates. Yet continual improvements in server efficiency kept total energy consumption … Read more