HBM Explained: Why High-Bandwidth Memory Is the Real Bottleneck in AI Chips

What is HBM?#

HBM is a JEDEC-standard kind of DRAM, the same memory family as a PC's main memory, arranged differently. Instead of separate chips spread across a circuit board, several DRAM dies are stacked on a base die and joined by through-silicon vias, vertical connections that run through the silicon itself. The finished stack sits next to the processor in the same package, which is what makes a bus 1,024 or 2,048 bits wide practical. Every accelerator in the table below uses it, from NVIDIA's Blackwell and Rubin GPUs and AMD's Instinct chips to Google's Ironwood TPU, AWS Trainium3 and Microsoft's Maia 200.

Why AI inference waits on memory, not math#

The roofline model, set out by Samuel Williams, Andrew Waterman and David Patterson at UC Berkeley in 2008, is a simple way to see why. A workload is capped by one of two ceilings: peak compute, in operations per second, or peak memory bandwidth, in bytes per second. What decides which ceiling applies is what the paper calls operational intensity, the operations a kernel does per byte it pulls from memory. Every chip has a ridge point where a workload flips from memory-bound to compute-bound. Below the ridge, a chip is limited by how fast it can stream bytes; above it, by how fast it can do math.

Large-model inference has two phases, and NVIDIA's own guide to LLM inference describes the split. Prefill, which reads the prompt, is a highly parallel matrix operation that keeps the GPU busy. Decode, which writes the answer one token at a time, is different: the speed at which weights, keys, values and activations move from memory to the GPU dominates latency, "not how fast the computation actually happens." A rough worked example (our arithmetic): a 70-billion-parameter model stored at 8 bits a parameter is about 70 GB of weights. On a GPU with 8 TB/s of HBM, reading those weights once takes about 8.75 milliseconds, so one sequence can get at most about 114 tokens a second however much compute the chip has. Doubling the FLOPS changes nothing in that sum. Doubling the bandwidth halves it.

How HBM is built#

An HBM stack is a pile of DRAM dies on a base die, wired together vertically with through-silicon vias. JEDEC's HBM4 standard, published in April 2025, allows stacks 4, 8, 12 or 16 dies high, with dies of 24 or 32 gigabits, for up to 64 GB in a 16-high stack. The base die at the bottom routes the stack's signals to the processor, and in HBM4 it has become a logic chip in its own right: Samsung builds its HBM4 base die on a 4nm logic process, and SK hynix was reported in 2024 to be moving its HBM4 base die to TSMC's logic process.

The stack then sits beside the processor on an interposer. In TSMC's original CoWoS-S design that is a large piece of silicon whose fine metal wiring links the logic chips and HBM stacks, with through-silicon vias carrying signals and power down to the package substrate. Silicon wiring is what allows the very dense connections a 2,048-bit bus needs. Ordinary graphics memory takes the opposite route. A GDDR chip sits farther away on the circuit board and drives each pin much faster: Samsung's 24Gb GDDR7, announced in October 2024, reaches 40 Gb/s per pin, against more than 9.2 Gb/s for Micron's HBM3E and the 8 Gb/s that JEDEC's HBM4 standard sets.

The arithmetic is simple. Bandwidth per stack is bus width times per-pin speed, divided by 8 to turn bits into bytes. HBM3E at 1,024 bits and about 9.2 Gb/s gives roughly 1.2 TB/s, Micron's figure. HBM4 at 2,048 bits and JEDEC's 8 Gb/s gives about 2 TB/s, the standard's headline number. At the more than 11 Gb/s Micron reports for its production HBM4, the same sum gives more than 2.8 TB/s. HBM wins by width at modest pin speeds, and more stacks on a package add more bandwidth again.

Where the HBM memory controller lives#

The HBM memory controller is not inside the stack. It is part of the processor that uses the memory. NVIDIA credits Rubin's near tripling of memory bandwidth over Blackwell partly to "new memory controllers", alongside "deep co-engineering with the memory ecosystem". JEDEC wrote HBM4 so that its interface stays backward compatible with existing HBM3 controllers, which lets one controller design work with both generations if a chipmaker needs it. The base die inside the stack is a separate job: it sits under the DRAM dies and passes their signals to and from that controller.

HBM3E versus HBM4#

HBM3E is the memory in most of the accelerators in the table below, and HBM4 went into production during 2026 at all three makers. Samsung said on February 12, 2026 that it had begun mass production of HBM4 and shipped commercial units. Micron said on March 16, 2026 that its 36 GB 12-high HBM4 was in high-volume production for NVIDIA's Vera Rubin. SK hynix told investors in July 2026, as reported by Edaily, that its HBM4 mass production began in the second quarter.

Read down the bandwidth column and the pattern is plain. The HBM3E and Ironwood rows sit between about 5 and 8 TB/s; the HBM4 rows sit above 20. NVIDIA says Rubin "nearly triples" Blackwell's memory bandwidth: up to 22 TB/s, against 8 TB/s for each Blackwell GPU in a DGX B200 (our division). AMD gets 23.3 TB/s from 12 stacks, which works out to about 1.9 TB/s a stack (our division), close to the JEDEC baseline of about 2 TB/s at 8 Gb/s. So on AMD's part the gain comes from HBM4's wider bus and its 12 stacks, not from faster pins.

Is packaging still the bottleneck?#

Putting HBM beside a processor needs advanced packaging, and TSMC's version is CoWoS (Chip on Wafer on Substrate). TSMC says CoWoS can build an interposer larger than twice the size of a lithography reticle, about 1,700 square millimeters, holding a large processor and several HBM stacks. That size is the problem. A silicon interposer bigger than one reticle has to be printed in pieces with mask stitching, which, as SemiWiki's 2026 comparison of the variants puts it, adds manufacturing complexity and defect risk as the interposer grows. CoWoS-L avoids one huge slab of silicon: it uses a redistribution-layer interposer for most of the wiring and embeds small silicon bridges, called local silicon interconnects, only where the densest links are needed, such as between processor and memory.

In 2023 and 2024, TSMC itself named packaging as the limit on AI chip supply. In September 2023, TSMC's then chairman Mark Liu said packaging, not the chip itself, was the holdup, and in July 2024 C.C. Wei said CoWoS capacity was still the bottleneck even after TSMC had more than doubled it in a year (both as reported by The Register). By August 2026 TSMC was telling a different story. Its advanced packaging vice president Jun He said CoWoS capacity had nearly doubled each year for three years and, in his words, "We haven't fully met demand yet, but we're getting very close." He named memory and ABF substrates as the key bottlenecks in the AI supply chain (TrendForce, citing Taiwanese press).

So, on TSMC's own account, the supply squeeze has moved from the package back toward the memory itself. The packaging story still matters, because HBM only works sitting beside the processor on an advanced package of this kind. But in 2026 HBM output matters at least as much as packaging slots or logic wafers.

How much of an AI chip's cost is HBM?#

By Epoch AI's estimate, HBM is now the largest single component cost in AI chips. Epoch, which averages component costs across the AI chips designed by NVIDIA, AMD, Google and Amazon, weighted by production volume, estimated in May 2026 that HBM rose from 52% of component spending in the first quarter of 2024 to 63% in the last quarter of 2025. Logic dies stayed near 13%, while packaging fell from about 19% to 15%. In money, Epoch put HBM spending across those four designers at roughly 12 billion dollars in 2024 and 32 billion in 2025, and expects the share to grow in 2026 as memory supply stays tight and prices rise. Counterpoint Research's HBM revenue table lists three suppliers: for the second quarter of 2026 it gives SK hynix 50%, Samsung 33% and Micron 18% (rounded), with SK hynix down from 58% a quarter earlier.

Can anything get around the memory wall?#

Several ideas chip away at the memory wall, each with a real limit. Processing-in-memory puts small compute units inside the memory so less data has to cross the bus: Samsung announced HBM-PIM, with an AI engine inside each memory bank, in February 2021, and SK hynix announced its GDDR6-AiM in February 2022. Neither is the main memory in any accelerator in the table above. CXL memory adds capacity over a link: CXL 3.0 runs at 64 GT/s on the PCIe 6.0 physical layer, which for a 16-lane link is roughly 128 GB/s each way before overhead (our arithmetic), against about 2 TB/s for a single HBM4 stack. It helps when a model or KV cache does not fit in HBM, and it is far slower when data spills into it. Chiplet standards such as UCIe let memory and compute be built and combined separately, but moving bits at HBM speed still needs a wide, short link, which still means stacking memory close to the processor.

The opposite bet keeps the working data in on-chip SRAM, which is far faster than HBM but holds far less. Cerebras' WSE-3, announced in March 2024, is a whole wafer with 44 GB of on-chip SRAM. NVIDIA's Groq 3 LPX, which NVIDIA introduced in March 2026 as a rack-scale inference accelerator for the Vera Rubin platform, gives each LPU 500 MB of SRAM at 150 TB/s; because that capacity is finite, NVIDIA spreads larger models across many of the 256 chips in a rack and pairs them with HBM-based Rubin GPUs, which keep prefill and decode attention. Even the HBM chips lean on SRAM: Google describes Ironwood's on-chip vector memory as a high-speed scratchpad and still warns that HBM "can still be a bottleneck" for memory-bound operations. SRAM works around the wall for some jobs. It has not replaced HBM for large models.

There is a deeper reason memory improves slowly. Imec, the Belgian research center, says dimensional scaling is slowing for DRAM too. Semiconductor Engineering reported in 2019 that scaling the storage capacitor in each cell is one roadblock, quoting Micron's Debra Bell: "In cell capacitor scaling, the aspect ratio is a challenge." Stacking more dies adds capacity, not bandwidth per die. The route to more bandwidth is a wider interface and faster signaling, which is exactly what HBM4's 2,048-bit bus does. Until something matches HBM's bandwidth at similar cost and power, the AI chip race is largely a contest over who gets the most out of a stack of DRAM dies beside the processor.