Memory Bandwidth, Not Compute, Limits Local LLM Speed

Memory bandwidth is the dominant factor in local LLM inference speed, not your GPU's TFLOP rating or your CPU's clock frequency. Understanding why requires looking at how transformers actually consume hardware resources.

Tokens Come From Moving Weights, Not Computing Them#

When a large language model generates a single token, it loads most of its weights from memory, runs a relatively small number of multiply-accumulate operations on them, and discards those weights until the next token. That pattern is the core problem.

Generating tokens is fundamentally memory-bound, not compute-bound. The relevant metric is arithmetic intensity: floating-point operations per byte of memory traffic. Training with large batches achieves high arithmetic intensity: weights are reused across many examples. Autoregressive inference at batch size one has extremely low arithmetic intensity: you load each weight, use it once or twice, and move on.

The theoretical peak TFLOPS printed on a product sheet become nearly irrelevant for single-user local inference.

The bottleneck is memory bandwidth: how fast weights travel from storage into compute units.

The Math Behind the Limit#

Think through the logic:

Divide model size by your device's memory bandwidth and you get a rough floor on per-token latency: the minimum time even with perfect utilisation. Doubling bandwidth roughly doubles token generation speed.

This is why a modest GPU with very high memory bandwidth consistently outperforms a compute-monster chip with slower memory at batch size one. The teraflops sit idle, waiting for data the memory bus cannot deliver fast enough.

Why Different Hardware Behaves So Differently#

Quantisation: Shrinking the Problem to Fit Your Pipe#

Quantisation trades model quality for inference speed through a pure memory bandwidth mechanism.

Lower precision means fewer bytes per parameter. Fewer bytes means less memory traffic per token. On fixed bandwidth, less traffic means higher throughput:

Methods like GGUF k-quants, GPTQ, and AWQ preserve the most important weight information while compressing aggressively. But speed gains flow through the same mechanism: reduced memory traffic. Quantisation helps most when you're deepest in the memory-bound regime, which is almost always true at local inference batch sizes.

KV Cache: A Second Memory Problem#

The key-value (KV) cache creates a second memory pressure that grows with context length. During inference, the model stores intermediate attention keys and values for every prior token. This cache lives in fast memory and is read on every forward pass.

Longer-context inference is slower not just because there is more computation, but because there is more memory traffic.

Practical Implications for Hardware Selection#

The Software Stack Sets How Close You Get to the Ceiling#

Hardware bandwidth is the ceiling; software determines how close you get. Inference runtimes like llama.cpp, MLX, and ONNX Runtime use different strategies (memory layout optimisations, custom kernels, asynchronous transfers) to saturate available bandwidth more efficiently. The same hardware can deliver meaningfully different token rates depending on how well the runtime exploits it.

Tiling strategies, fused kernels, and prefetching exist to hide memory latency and maximise bandwidth utilisation. Choosing a well-optimised runtime matters, particularly on novel hardware where generic code paths may leave significant bandwidth untapped.

---

When evaluating any local-AI hardware claim, the first question to ask is not "how many TFLOPS?" but "how wide is the memory pipe, and can the model fit inside it?"