Memory Bandwidth, Not Compute, Limits Local LLM Speed

Memory bandwidth is the dominant factor in local LLM inference speed, not your GPU's TFLOP rating or your CPU's clock frequency. Understanding why requires looking at how transformers actually consume hardware resources.

Tokens Come From Moving Weights, Not Computing Them#

When a large language model generates a single token, it loads most of its weights from memory, runs a relatively small number of multiply-accumulate operations on them, and discards those weights until the next token. That pattern is the core problem.

Generating tokens is fundamentally memory-bound, not compute-bound. The relevant metric is arithmetic intensity: floating-point operations per byte of memory traffic. Training with large batches achieves high arithmetic intensity — weights are reused across many examples. Autoregressive inference at batch size one has extremely low arithmetic intensity — you load each weight, use it once or twice, and move on.

The consequence: theoretical peak TFLOPS printed on a product sheet become nearly irrelevant for single-user local inference. Your bottleneck is memory bandwidth — how fast weights travel from storage into compute units.

The Math Behind the Limit#

Think through the logic:

• A 7-billion-parameter model at 4-bit precision occupies roughly 3.5 GB.
• A 70-billion-parameter model at 4-bit is around 35 GB.
• To generate one token, essentially the entire set of active weights must be read from memory once.

Divide model size by your device's memory bandwidth and you get a rough floor on per-token latency — the minimum time even with perfect utilisation. Doubling bandwidth roughly doubles token generation speed.

This is why a modest GPU with very high memory bandwidth consistently outperforms a compute-monster chip with slower memory at batch size one. The teraflops sit idle, waiting for data the memory bus cannot deliver fast enough.

Why Different Hardware Behaves So Differently#

Discrete consumer GPUs typically use GDDR memory offering moderate bandwidth, with VRAM capacities that may not fit larger models. When a model exceeds VRAM, inference spills to system RAM — and the PCIe bus is orders of magnitude narrower than VRAM bandwidth. Token generation slows dramatically.

High-end discrete GPUs use HBM (High Bandwidth Memory), a stacked architecture providing dramatically higher bandwidth than GDDR, plus tens or hundreds of gigabytes of VRAM. They dominate server inference not primarily because they compute faster, but because they move weights faster.

Apple Silicon (M-series) uses unified memory where the CPU, GPU, and Neural Engine share the same physical pool over a very wide internal bus. Bandwidth is substantially higher than typical discrete-GPU GDDR, and the absence of a PCIe bottleneck allows surprisingly large quantised models to run with respectable token rates. The trade-off: memory capacity is capped by what's soldered to the board.

Dedicated AI accelerators push this logic further — designed to maximise memory bandwidth per watt for transformer inference, often sacrificing general-purpose flexibility that GPUs provide.

Quantisation: Shrinking the Problem to Fit Your Pipe#

Quantisation trades model quality for inference speed through a pure memory bandwidth mechanism.

Lower precision means fewer bytes per parameter. Fewer bytes means less memory traffic per token. On fixed bandwidth, less traffic means higher throughput:

• FP16 (16-bit): Full fidelity, maximum bytes, slowest on given hardware.
• Q8 (8-bit): Half the bytes of FP16, typically ~2× token rate with small quality loss.
• Q4 (4-bit): Quarter the bytes of FP16, can double token rate again, with noticeable quality degradation depending on the method.
• Q2–Q3: Extremely compact, but quality often degrades enough to matter.

Methods like GGUF k-quants, GPTQ, and AWQ preserve the most important weight information while compressing aggressively. But speed gains flow through the same mechanism: reduced memory traffic. Quantisation helps most when you're deepest in the memory-bound regime — which is almost always true at local inference batch sizes.

KV Cache: A Second Memory Problem#

The key-value (KV) cache creates a second memory pressure that grows with context length. During inference, the model stores intermediate attention keys and values for every prior token. This cache lives in fast memory and is read on every forward pass.

As context length grows, KV cache grows linearly with it. For long contexts, KV cache memory can dwarf the model weights themselves, creating compounding demands on bandwidth: loading weights and loading/storing KV cache simultaneously.

Longer context inference is slower not just because of more computation, but because of more memory traffic. Hardware with higher bandwidth degrades more gracefully as context grows. Strategies like KV cache quantisation and sliding-window attention exist partly to manage this bandwidth pressure.

Practical Implications for Hardware Selection#

Prioritise bandwidth over compute. When comparing two GPUs, check memory bandwidth alongside VRAM capacity. Higher bandwidth with the same VRAM often serves inference better.

VRAM capacity sets the ceiling. A model forced onto slower memory (system RAM) stalls regardless of fast-memory bandwidth. Capacity and bandwidth together define what's feasible.

Unified memory eliminates one bottleneck. A large, fast, unified pool avoids the PCIe spill problem — a genuine architectural advantage for local inference, not marketing.

Batch size changes the calculus. If running a local API server handling multiple concurrent requests, higher batch sizes raise arithmetic intensity and bring compute back into play. Memory-bound logic is strongest for interactive single-user generation.

System RAM and CPU bandwidth matter for CPU inference. Running entirely on CPU (llama.cpp without a capable GPU), CPU memory bandwidth and RAM speed become the direct bottleneck. DDR5 systems with high-frequency memory have a meaningful edge over older DDR4 machines.

The Software Stack Sets How Close You Get to the Ceiling#

Hardware bandwidth is the ceiling; software determines how close you get. Inference runtimes like llama.cpp, MLX, and ONNX Runtime use different strategies — memory layout optimisations, custom kernels, asynchronous transfers — to saturate available bandwidth more efficiently. The same hardware can deliver meaningfully different token rates depending on how well the runtime exploits it.

Tiling strategies, fused kernels, and prefetching exist to hide memory latency and maximise bandwidth utilisation. Choosing a well-optimised runtime matters, particularly on novel hardware where generic code paths may leave significant bandwidth untapped.

---

When evaluating any local-AI hardware claim, the first question to ask is not "how many TFLOPS?" but "how wide is the memory pipe, and can the model fit inside it?"