Quantization Explained: What Q4, Q8, and FP16 Actually Do to a Local Model

Quantization is the process of reducing the numerical precision of a model's weights, and it is the single most important lever you have when running large language models locally. The format you choose, Q4, Q8, or FP16, determines how much memory you need, how fast tokens generate, and how much quality you give up. If you have ever downloaded a GGUF file and stared at names like Q4_K_M or Q8_0 wondering what to actually pick, this is the complete answer.

What a model weight actually is#

Every parameter in a neural network is a number, a weight learned during training. During training those numbers are stored in high precision, almost always BF16 (16-bit brain float) on modern hardware. A 7-billion-parameter model in 32-bit float consumes around 28 GB just for the weights. The core insight of quantization is that most of those weights do not need that precision to produce good outputs. Quantization maps the original high-precision values onto a smaller set of discrete levels, then stores the index into that set rather than the full float. The trade-off is always the same shape: fewer bits means less memory, faster inference, and more quality loss. The only question is how steep that curve is and where you should sit on it.

The precision tiers, and what they cost#

FP16 is what most local runners treat as full quality: two bytes per weight, so a 7B model lands near 14 GB, a 70B model well past 130 GB. Q8 halves that to one byte per weight (a 7B model near 8 GB) and is, for most tasks, indistinguishable from 16-bit, with perplexity typically rising well under one percent. Q4 drops to four bits, roughly 4 to 5 GB for a 7B model, which is where local inference becomes accessible on ordinary hardware. Slide your model size below and watch where each tier lands against your memory:

Q4 and the variant names that matter#

With Q4 the variant naming matters enormously. Q4_0 is a naive implementation, a single scale per 32-weight block, fast but relatively lossy. Q4_K_M and Q4_K_S, the K-quants, group weights into hierarchical super-blocks and quantize the scale factors themselves, freeing up bits to spend where they matter, and they mix precision across the model: sensitive tensors (attention and the output and embedding layers) stay at higher bit depth while less sensitive feed-forward layers are pushed harder. The result is that Q4_K_M recovers a large chunk of the quality Q4_0 loses at nearly the same file size. A common myth worth killing: the K does not stand for k-means. K-quants use ordinary affine quantization; the name traces to the method's author, Iwan Kawrakow. What makes them better is the super-block structure and the mixed bit allocation, not any exotic math.

Why block structure beats bit depth#

Smaller quantization blocks adapt to local weight statistics better but spend more bits on scale-factor metadata; larger blocks compress more but add error. K-quants resolve the tension with two levels: a 256-weight super-block subdivided into sub-blocks of 32, where each sub-block scale is itself quantized against a super-block scale. That is how a format spends almost no extra bits on metadata yet still adapts tightly, and it is why Q4_K_M punches above its bit depth. Reading the suffixes: _S, _M, and _L refer to the mix of precision across tensor types, so _M keeps more of the sensitive tensors high and lands slightly larger but higher quality than _S. A newer wrinkle is importance-matrix (imatrix) quantization: the tooling runs a small calibration set through the model to find which weights actually move the outputs, then protects those from aggressive rounding. Most good low-bit GGUFs today are imatrix-calibrated, and some ship dynamic quants that upcast individually chosen layers, which is why a modern 4-bit release holds together far better than a naive Q4 from a couple of years ago.

Memory, speed, and the CPU fallback#

Memory is the primary reason to quantize, but speed is a real secondary benefit. On a GPU, quantized weights are typically dequantized back to 16-bit right before the matrix multiplies, block by block; the speed gain comes not from doing less math but from fitting more of the model on-chip, because VRAM bandwidth dwarfs the CPU-to-GPU transfer path. A model that fits entirely in VRAM runs many times faster than one that spills to system RAM. On CPU-only inference the math itself can run in integer arithmetic, which is where you see genuine compute speedups, and running a Q4 model on CPU is genuinely viable for a 7B to 8B model where 16-bit would not be. Do not forget the KV cache: at long context it can rival the weights for memory, and quantizing the cache too (llama.cpp supports Q8 and Q4 cache) buys back headroom for a small quality cost, a separate lever from weight quantization.

One practical rule ties it together: a bigger quantized model usually beats a smaller full-precision one. A well-trained 27B model at Q4 will typically outperform an 8B model at FP16 on real tasks. Quantization compresses capability, it does not destroy it, so if your bottleneck is memory rather than compute, spend your gigabytes on more parameters at four bits before you spend them on more precision at fewer parameters.