AI inference is a wall of matrix math. GPUs win because their thousands of cores, dedicated matrix units, and terabytes-per-second of memory bandwidth map onto that math almost perfectly, while CPUs, built for flexibility and low-latency single threads, choke on the parallelism.
A deep read — the full picture, with the receipts.
AI inference, running a trained model on new input to produce a prediction, a classification, or the next token of text, is underneath the marketing a wall of linear algebra. With models now measured in hundreds of billions of parameters and served to millions of requests, the hardware you run that math on decides your latency, your throughput, and your bill. Traditional CPUs earn their keep on general-purpose work because they are flexible and fast at any single task. But inference is not a general-purpose task. It is the same simple operation repeated across enormous arrays of numbers, and that is exactly the shape of problem GPUs were built to eat.
The architectural divide#
CPUs and GPUs are built around opposite bets. A CPU spends most of its transistor budget making a handful of powerful cores as fast as possible on one thing at a time: deep pipelines, out-of-order execution, branch prediction, and large caches all exist to minimize latency and handle unpredictable, branchy code. A GPU makes the reverse bet. It was originally built to render graphics, a problem that is parallel by nature, so instead of a few complex cores it packs thousands of simpler ones, all executing the same instruction across different data at once. Each core is individually less clever, but when the job is apply this identical operation to a million numbers, ten thousand modest cores crush a few brilliant ones.
For inference math: two opposite bets
A few brilliant cores
CPU
- Dozens of complex cores tuned for low latency on one task at a time.
- Deep pipelines, branch prediction, and big caches for unpredictable, branchy code.
- Vector units (AVX-512) offer a handful of parallel lanes per core, not thousands, and no Tensor Core equivalent.
- Wins on web servers, databases, transaction processing, and the glue around the model.
Thousands of simple cores
GPU
- Thousands of simple cores running the same instruction across different data (the SIMT model).
- Dedicated matrix hardware (Tensor Cores, Matrix Cores) does fused multiply-accumulate over a whole tile at once.
- Low-precision arithmetic (FP16, BF16, FP8, even FP4) moves more numbers per clock, and inference tolerates it.
- High-bandwidth memory feeds the cores at terabytes per second so they never starve.
The math is the whole reason#
Strip a neural network down and it is mostly matrix multiplication. A layer takes an input vector, multiplies it by a weight matrix, adds a bias, and applies a nonlinearity; attention layers and convolutions are the same story in different arrangements. These are dense, regular operations over large arrays, and every output element is a sum of products that does not depend on the others. That independence is the whole game: a GPU can assign thousands of threads to compute thousands of those products simultaneously. A CPU has to lean on scheduling tricks and sheer clock speed to keep up, which adds overhead and still leaves it an order of magnitude or more behind on raw throughput.
GPU cores, vs dozens on a server CPU
GPU memory bandwidth (HBM)
GPU-to-CPU memory-bandwidth gap
Lowest inference precision (Blackwell)
Feeding the cores: memory, not just compute#
Raw compute is only half the story; you have to feed it. Every one of those thousands of cores needs data, and if memory cannot supply it fast enough, the cores sit idle. A data-center GPU is bonded to high-bandwidth memory (HBM) stacked right next to the compute die, delivering multiple terabytes per second. A high-end server CPU, reading from ordinary DDR system RAM, gets hundreds of gigabytes per second, often a 10x-plus gap. GPU hardware warp schedulers hide memory latency by instantly switching to another group of threads whenever one is waiting on data, keeping the arithmetic units busy. A CPU's memory system is tuned for the opposite problem: low-latency access across a modest number of independent threads.
Inference is two phases with opposite bottlenecks#
Inference applies a frozen model to new input, and for a large language model it splits into two phases. Prefill, processing the whole prompt at once, is compute-bound: it is one big batch of matrix multiplies, so it leans on the GPU's raw FLOPS. Decode, generating the answer one token at a time, is memory-bandwidth-bound: each new token requires reading the model's weights and the growing KV cache back out of memory, and there is not enough arithmetic per byte to keep the compute units saturated. That is why HBM bandwidth matters so much for serving speed. The GPU wins prefill on compute and decode on bandwidth; a CPU, starved for bandwidth and lacking the parallel width, loses on both counts.
The costs, and why it is rarely GPU-or-CPU#
None of this is free. Data-center GPUs are expensive to buy, draw far more power than CPUs, run hot enough that dense deployments increasingly rely on liquid cooling, and live in a fast-moving software stack (CUDA, drivers, vLLM, TensorRT) that needs upkeep. Blackwell-class systems are sold as full rack-scale machines because at the high end the unit of deployment is no longer a single card but an interconnected cluster. CPUs remain indispensable even inside a GPU-centric inference server: they handle orchestration, request routing, tokenization, and everything that is not the matrix math, and for small models or low volumes a CPU alone can be the cheaper, simpler choice. The practical setup is a division of labor, not a contest.
Why are GPUs faster than CPUs for AI inference?
Because inference is mostly dense matrix multiplication: the same operation applied independently across huge arrays. A GPU has thousands of simple cores plus dedicated matrix hardware to do that in parallel, and terabytes-per-second of memory bandwidth to feed them. A CPU has dozens of complex cores tuned for low-latency, branchy work, so it falls an order of magnitude or more behind on this specific shape of problem.
Do you still need a CPU if you have a GPU?
Yes. Even a GPU-centric inference server relies on the CPU for orchestration, request routing, tokenization, data loading, and all the branchy logic around the model. The GPU does the dense parallel math; the CPU runs everything else. For small models or low request volumes, a CPU alone can even be the cheaper, simpler choice.
Why does memory bandwidth matter as much as compute for inference?
Generating text one token at a time (the decode phase) is memory-bandwidth-bound: each token requires reading the model's weights and the growing KV cache out of memory, and there is not enough arithmetic per byte to keep the compute units busy. A GPU's high-bandwidth memory, at multiple terabytes per second, is what keeps decode fast, which is why it is as central to the GPU's advantage as its raw FLOPS.
Are GPUs always the right choice for running a model?
No. GPUs win when you are serving large models at scale with real throughput and latency demands. For branchy, latency-sensitive, single-threaded work, or for small models at low volume, a CPU is often the right and cheaper tool, and forcing that work onto a GPU wastes money. The honest evaluation weighs the workload's shape against each architecture's strengths and against power, cooling, and engineering cost.
Discussion