The 3 Markets in an AI Trench Coat
Year: 2026
Type: other
Abstract: Episode summary: Most people still think the answer to AI hardware is "buy more GPUs." But the actual landscape has fragmented into three or four different markets. This episode explores how training and inference have diverged so completely that the optimal chip for one is increasingly wrong for the other. We break down the rigidity spectrum — from Google's custom TPUs to Groq's LPUs to NVIDIA's new heterogeneous architectures — and explain why the type of inference you're doing (batch processing, interactive chat, or agentic swarms) should determine your hardware strategy. Plus: what the $20 billion Groq acquisition tells us about the future of inference silicon. Show Notes The AI hardware market is no longer one market. It's three or four different markets operating under the same label, and the hardware that's optimal for training is increasingly wrong for inference. **The Fundamental Divide: Training vs. Inference** Training a model is a cost center. Inference is the product. You train once, but you run inference millions of times a day. The economics are completely different. Training cares about raw teraflops and throughput; inference cares about cost per query and latency consistency. An H100 GPU draws over 700 watts per chip and relies on complex memory hierarchies with variable latency — fine for training, expensive overhead for serving a chatbot that needs sub-100-millisecond responses. **The Rigidity Spectrum** Different hardware architectures represent different points on a spectrum from flexibility to efficiency. On one end: ASICs (application-specific integrated circuits), purpose-built for a specific model or model class. A recent paper from XgenSilicon used reinforcement learning to co-design a chip and compiler for Llama 3.1 8B, achieving nearly 30,000 tokens per second. But that chip runs that model and nothing else. The non-recurring engineering cost is enormous — you need Google-scale volume to justify it. In the middle: LPUs (language processing units), pioneered by Groq and now owned by NVIDIA for $20 billion. LPUs use on-chip SRAM with no external memory, no caches, and no variable latency. If the compiler says an operation takes 28.5 milliseconds, it takes exactly 28.5 milliseconds every time. The tradeoff: on-chip SRAM is limited to about 230 MB, so large models must be sharded across hundreds of chips. On the flexible end: GPUs, which can run anything but leave efficiency on the table. **Heterogeneous Architectures: The Emerging Paradigm** NVIDIA's post-acquisition strategy reveals where the industry is heading. Their new LPX system, co-designed with the Vera Rubin NVL72 platform, creates a heterogeneous architecture where different parts of a single inference request get routed to different processors in real time. Rubin GPUs handle prefill and decode attention; LPUs accelerate the latency-sensitive feed-forward and mixture-of-experts decode. The Dynamo orchestration software claims up to 35x higher inference throughput per megawatt. The bottleneck for many inference workloads has shifted from compute to data movement. The LPX system delivers 40 PB/s of on-chip SRAM bandwidth — orders of magnitude more than an H100's 3.35 TB/s of HBM bandwidth. **How Inference Type Dictates Hardware** Batch processing (classifying millions of documents overnight) favors GPUs — you can batch queries, hide latency, and maximize utilization. Interactive chat (an AI coding assistant) requires deterministic low latency — LPUs excel here. Agentic AI (multi-agent swarms with tight tool-calling loops) pushes latency requirements to the extreme, which is why NVIDIA frames the LPX as "the agentic inference engine," targeting 1,000 tokens per second per user. Even edge inference is fragmenting. NPUs (neural processing units) deliver up to 60% faster inference than GPUs while using 44% less power on matrix-vector workloads — critical for battery-constrained devices, though GPUs remain superior for large-dimension matrix multiplication. **The Strategic Question** Most organizations can't design custom ASICs or deploy hundreds of LPUs. GPUs remain the pragmatic default, but the definition of "GPU" is expanding as NVIDIA builds heterogeneous systems designed to keep customers in their ecosystem. The lock-in concern is real: CUDA, TPU software stacks, Inferentia SDKs, and LPX toolchains all require different engineering investments. The coupling between model and hardware is a strategic question, not just a technical one. Tight coupling gives efficiency but low flexibility and high switching costs. Loose coupling gives flexibility but leaves performance on the table. Where you land depends on your scale, workload predictability, and tolerance for vendor lock-in. Listen online: https://myweirdprompts.com/episode/ai-hardware-inference-coupling
Institution: Google DeepMind (United Kingdom)
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7155628758