HBM Explained: Why High-Bandwidth Memory Is the Real Bottleneck in AI Chips

Why the memory sets the speed, not the math#

High-bandwidth memory is the hidden governor on every AI accelerator shipping today. Not the matrix engines, not the transistor count. The moment you run a large language model in inference, the dominant cost is almost never arithmetic. It is the time spent waiting for weights to arrive from memory. Understanding why cuts through most of the AI-chip marketing noise.

The roofline model is the tool that makes this clear. It says a workload is bounded by one of two ceilings: peak compute throughput, measured in floating-point operations per second, or peak memory bandwidth, measured in bytes per second. The ratio between the arithmetic a kernel does and the data it must pull from memory to do it is called arithmetic intensity, expressed as operations per byte. Every chip has a ridge point, the intensity at which a workload flips from memory-bound to compute-bound. Below the ridge you are limited by how fast you can stream bytes; above it, by how fast you can do math.

Modern accelerators have very high ridge points, because their compute throughput has scaled far faster than their memory bandwidth. Only the most compute-dense operations, like the large batched matrix multiplications in training, live above the ridge. Autoregressive token generation sits well below it: each new token loads the model's weight tensors, tens to hundreds of gigabytes, through the memory subsystem once per forward pass, and does comparatively little math on them. The chip becomes an expensive memory bus with a math unit attached. Doubling its FLOPS on that workload accomplishes almost nothing if bandwidth stays flat.

How HBM is actually built#

HBM is not a faster version of an ordinary memory module. It is a different physical arrangement. A conventional GDDR chip sits beside the processor on a circuit board and connects over a relatively narrow wire interface. HBM stacks DRAM dies directly on top of each other, typically eight to twelve high, and connects the layers with through-silicon vias: thousands of tiny copper pillars etched vertically through each die, forming a dense electrical highway between them. At the bottom of the stack sits a base logic die that routes all those connections.

The assembled stack is then placed on a silicon interposer, a passive slab of silicon carrying fine-pitch wiring, right next to the accelerator die. The interposer joins the two with an interface measured in thousands of signal wires, which is what enables bandwidth that GDDR cannot approach at comparable power. It is also what makes fitting several stacks alongside one accelerator physically possible, and it introduces its own constraint, which the supply section returns to.

The critical point is what this buys. Not clock-rate speed, since HBM actually runs at lower per-pin data rates than GDDR, but width. Bandwidth is bus width multiplied by frequency multiplied by data rate. HBM wins by making the bus enormously wide, an HBM3E stack presents a 1,024-bit interface, while keeping voltage low so power stays manageable. More stacks, more total bandwidth. The newest generation doubles that width again, which is where the generations diverge.

The generational ladder#

Each generation solved a specific problem. HBM2 established the format: multiple stacks with enough bandwidth to feed the first serious deep-learning accelerators. HBM2E raised the data rate and per-stack capacity on the same physical interface, which mattered as models grew. HBM3 crossed a real threshold, lifting both bandwidth and capacity together, the combination that helps inference most, since more data resident in the stacks means fewer trips to slower memory and faster delivery means the math units starve less. HBM3E is the current volume generation, and HBM4 is the one ramping into production now. The interesting split is between those last two.

Notice what improves each time. The headline gain is width and signaling, not clock speed. HBM4's jump to a 2,048-bit interface with 32 channels is the clearest case: the format keeps winning bandwidth by getting wider and running more channels in parallel, while keeping per-pin voltage low so power stays in budget. That is the opposite of how a processor chases performance, and it is the only way to move the volume of data a modern accelerator needs without cooking the package. Nvidia's Rubin generation is the first major accelerator built around HBM4, carrying up to 288 GB at up to 22 terabytes per second per GPU, close to three times what its Blackwell predecessor delivered on HBM3E.

The real chokepoint is packaging#

The packaging that co-locates HBM stacks with an accelerator die on a silicon interposer is made at only a handful of facilities. TSMC's version is called CoWoS, short for Chip on Wafer on Substrate. The interposer itself is processed on an older, cheaper node, but the assembly is precise, its yield is low relative to ordinary chip packaging, and the lines that do it are a separate, capital-heavy investment with long buildout times.

You cannot fix a CoWoS shortage by buying more wafer starts. When AI-accelerator demand surged, the binding limit on how many Blackwell GPUs or MI350s could ship was not TSMC's leading-edge logic capacity, it was CoWoS capacity. The same silicon could be fabbed; it could not be packaged fast enough. The largest packages have already outgrown a single lithography reticle's worth of interposer, forcing stitched variants, TSMC's CoWoS-L, that build the interposer from multiple pieces. More capability, but more assembly steps and more ways to lose yield.

So the supply choke for AI compute is not really a chip story. It is a packaging story. TSMC has expanded CoWoS capacity aggressively and demand has still outrun it. Hyperscalers buying compute at scale are competing for packaging slots as much as for wafer allocations, a subtler constraint than the usual GPU-shortage framing, and a more accurate one.

Why HBM dominates the bill of materials#

HBM is costly to make. The yield challenges of forming thousands of through-silicon vias, stacking a dozen dies, and assembling them on an interposer mean it carries a large premium over conventional memory of the same raw capacity. By analyst estimates the memory can account for something like a third to more than 40 percent of a high-end accelerator's total bill of materials, which puts the memory subsystem in the same cost bracket as the logic die it feeds. Supply is concentrated too: SK Hynix, Samsung, and Micron are the only firms making HBM in volume. SK Hynix led on yield through the HBM3E era, the HBM4 round narrowed the field, with Micron reaching high-volume production for Nvidia's Rubin platform and all three vendors shipping. New demand still cannot summon new supply quickly, because HBM needs dedicated DRAM tooling and new capacity takes years to build.

Every serious AI chip hits the same wall#

Several approaches chip away at the constraint, each with a real limit. Processing-in-memory, which Samsung and SK Hynix have both prototyped, puts small compute elements inside the memory so less data has to cross the interface at all; it helps a narrow set of operations and is hard to program generally. CXL-attached memory pools add capacity over a coherent link, which relieves pressure when a model or KV cache will not fit in HBM, but CXL bandwidth is far below HBM's, so anything that spills into it hits a cliff. Die-to-die chiplet interconnects like UCIe let memory and compute be built and assembled separately, improving flexibility and yield economics without changing the physics: moving bits at HBM speed still needs a wide, short, low-voltage link, which still means something stacked close on an interposer. These extend around HBM. They do not replace it.

None of this is specific to one company. AMD's MI350 stacks HBM3E on an advanced package; Google's TPUs, through Ironwood, use HBM and hit the same roofline; Amazon's Trainium and Microsoft's Maia navigate the identical trade. Two chipmakers took the opposite bet: Cerebras put a whole system on one wafer-scale piece of silicon to avoid off-chip memory traffic, and Groq keeps weights in on-chip SRAM. Both trade the memory wall for other limits, since on-chip SRAM is far faster but far smaller and costlier per bit, and wafer-scale integration has its own yield and size ceilings. You can architect around the bottleneck for specific workloads; you cannot escape the bandwidth-to-compute ratio for the workloads that dominate inference at scale.

There is a deeper reason the memory side improves slowly. DRAM cells have not shrunk as fast as logic over the past decade, because the tiny capacitor that stores each bit gets harder to build reliably as geometries shrink. Stacking more dies buys capacity, not bandwidth per die. The path to more bandwidth is a wider interface and better signaling, which is exactly what HBM4's 2,048-bit bus does, rather than smaller features. That keeps the memory improvement curve flatter than AI compute demand would prefer, which is the whole reason this is a bottleneck and not a footnote. Until a replacement can match HBM's bandwidth density at comparable cost and power in a similar-size package, the AI-chip race is, at bottom, a contest over who gets the most out of a stacked pile of DRAM dies on a slab of passive silicon.