Advanced Technology

Inference 2.0: How enterprise AI is reshaping AI system architectures

by

Predictive, generative, and agentic AI are not successive generations. They coexist inside enterprise systems, each placing fundamentally different demands on the underlying compute infrastructure. The enterprise context layer A consumer chatbot relies primarily on pretraining weight memory and optional web search. An enterprise system must operate against information absent from pretraining: engineering specifications, financial ledgers, … Read more

Enterprise AI: Reshaping AI processor architectures

by

For the past decade, the artificial intelligence landscape followed a remarkably simple recipe: if you wanted better AI, you built larger models, fed them more data, and added more GPUs. The formula worked, and it worked spectacularly. Every new generation of hardware enabled larger neural networks, and larger neural networks broadened their reach. What began … Read more

MLPerf and the rise of latency-aware LLM benchmarking

by

Any discussion of modern AI system performance must include MLCommons and its MLPerf benchmark suite, which has become the industry’s de facto standard for measuring machine learning performance. Since its debut in 2018, MLPerf has provided a neutral, peer-reviewed framework for comparing hardware and software platforms across a broad range of AI workloads. The original … Read more

The hidden bottleneck in LLM inference and the impact on MLPerf benchmarking

by

Recent frontier LLM inference benchmarks have highlighted a recurring pattern. GPU-based systems deliver outstanding throughput when latency is not a concern, but their performance drops sharply once real-time response requirements are imposed. This behavior is sometimes attributed to software inefficiencies or suboptimal system tuning. In reality, the root cause lies much deeper. It reflects a … Read more

GPUs: A high-throughput architecture confronting a workload shift

by

There is a growing architectural tension at the heart of modern AI infrastructure. The processors that enabled the deep learning revolution—graphics processing units (GPUs)—remain the dominant engines of large-scale training and inference. Yet the computational profile of frontier language models is evolving in ways that increasingly expose the structural assumptions embedded in GPU design. Memory … Read more

The truth about AI inference costs: Why cost-per-token isn’t what it seems

by

The AI industry has converged on a deceptively simple metric: cost per token. It’s easy to understand, easy to compare, and easy to market. Every new system promises to drive it lower. Charts show steady declines, sometimes dramatic ones, reinforcing the impression that AI inference is rapidly becoming cheaper and more efficient. But simplicity, in … Read more

From Data to Drain: How AI Is Devouring the World’s Electricity

by

America’s data centers managed to keep their electricity use surprisingly steady from 2005 to 2017, with rather small annual increments contained via constant improvement in electronics. Then, around 2017, AI arrived forcefully and disrupted that stability. AI required a different computing machine, one designed not for ordinary tasks, such as our beloved PC, but for … Read more

Why memory swizzling is hidden tax on AI compute

by

Walk into any modern AI lab, data center, or autonomous vehicle development environment, and you’ll hear engineers talk endlessly about FLOPS, TOPS, sparsity, quantization, and model scaling laws. Those metrics dominate headlines and product datasheets. If you spend time with the people actually building or optimizing these systems, a different truth emerges: Raw arithmetic capability … Read more