How to choose the right quantization for a local LLM

The short answer: start with Q4_K_M, and move up to Q5_K_M only when the bigger file still leaves room for your context. On Llama 3.1 8B, the Q5_K_M file is 0.81 GB larger (5.73 GB against 4.92 GB on Hugging Face), and llama.cpp's own table puts its perplexity loss at about a third of Q4_K_M's (+0.0569 against +0.1754 on Llama 3 8B). Both losses are small, so our rule is to spend spare memory on a bigger model before a fifth bit.

By the end of this tutorial you will be able to look at a list of model files like Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ4_XS and pick the one that fits your hardware and your quality bar on purpose, not by guessing. You need no special tools, only a local LLM in a quantized file format such as GGUF and a rough idea of how much RAM or VRAM you can spare.

Step 1: Understand what quantization is#

A model's weights are numbers. By default they are stored as 16-bit or 32-bit floating-point values. Quantization stores those same weights using fewer bits each: 8 bits, 5 bits, 4 bits, sometimes fewer. Lower precision means each weight takes less memory and moves through the processor faster.

That is why a quantized model loads on a laptop that could never hold the full-precision version.

The tradeoff is rounding error. Squeezing a weight into 4 bits throws away information, and that error accumulates across billions of parameters.

Done well, the loss is small and barely shows up in output quality. Done too aggressively, the model starts producing weaker, less coherent text.

llama.cpp's documentation measures that loss in perplexity (and KL divergence). Perplexity is how well the model predicts a test text, where lower is better. Every quality number in this guide is the perplexity a quant adds, as llama.cpp's quantize tool lists it.

Step 2: Read the labels#

GGUF is the model file format llama.cpp uses, and the quant name is in the file name. Take Q4_K_M (you will also see it written q4km, or Q4_K, which llama.cpp's quantize tool accepts as an alias for Q4_K_M):

So Q4_K_M is a 4-bit k-quant with a medium mix, and Q5_K_M is the same recipe at 5 bits. Q6_K is a 6-bit k-quant. Q8_0 is an older, plain format: blocks of 32 weights with one scale each, at 8.5 bits per weight. Names like Q4_K_L or Q6_K_L are not llama.cpp types. They come from the uploader bartowski, whose Qwen3 32B model card describes them as the standard quant with the embedding and output weights kept at Q8_0.

Q4_K_M vs Q5_K_M vs Q6_K vs Q8_0, side by side#

The sizes below are the real file listings of three Hugging Face GGUF repositories by bartowski, read on 2026-10-19. GB per billion parameters is the file size divided by the parameter count Hugging Face reports for each repo (8.03B, 14.77B and 32.76B). The perplexity column is copied from llama.cpp's quantize tool, which measures it on Llama 3 8B.

Two things stand out. The perplexity cost is small from Q8_0 down to Q4_K_M and then climbs steeply below it: Q3_K_M adds almost four times what Q4_K_M does, and Q2_K adds twenty times as much. And GB per billion barely moves across these three models, from 8B to 32B, so for a dense model the parameter count alone gives you a close size.

Q4_K_M vs Q5_K_M#

Q5_K_M costs about 16 to 18% more memory: 0.81 GB more on the 8B, 1.51 GB on the 14B, 3.45 GB on the 32B. For that, llama.cpp's table cuts the added perplexity from +0.1754 to +0.0569. Pick Q5_K_M when it fits with your context to spare and Q4_K_M when it does not. Our rule: never drop to a smaller model just to afford the fifth bit.

Q4_K_M vs Q4_K_S#

Q4_K_S saves very little: 0.23 GB on the 8B and 0.99 GB on the 32B. It adds about half again as much perplexity (+0.2689 against +0.1754). Use it only when Q4_K_M misses your memory by less than a gigabyte.

Q6_K vs Q8_0#

On paper Q8_0 is the closer copy (+0.0026 against +0.0217), but both add almost nothing in llama.cpp's table, and Q6_K is about 23% smaller: 1.94 GB less on the 8B, 7.93 GB less on the 32B. In llama.cpp's README speed table for Llama 3.1 8B, Q6_K also generated text faster (58.67 against 50.93 tokens per second on its test machine). Our call: Q6_K, unless you are comparing quants against the original.

Q4_K_M vs Q8_0#

Q8_0 is about 1.75 times the size of Q4_K_M on every model in the table. In the same README table it generated about 29% fewer tokens per second (50.93 against 71.93); llama.cpp's k-quant PR notes that inference speed tends to follow model size. If the memory Q8_0 needs would fit a larger model at Q4_K_M, the larger model is our pick.

IQ quants (IQ4_XS, IQ3_M, IQ2_M) explained#

The IQ types (i-quants) are a newer family, added to llama.cpp from January 2024. Hugging Face's GGUF documentation describes each IQ type as using a super-block scale plus an importance matrix ("imatrix"). llama.cpp added the imatrix the same month: statistics gathered by running sample text through the model, used to weight which rounding errors matter most. bartowski's Qwen3 32B repo ships the Qwen_Qwen3-32B.imatrix file and says every quant in it was made with the imatrix option.

• IQ4_XS is a 4.25 bits-per-weight base type. On Qwen3 32B it is 2.07 GB smaller than Q4_K_M, which is room for about 7,900 more tokens of context. bartowski's model card calls it "smaller than Q4_K_S with similar performance".
• IQ3_M and IQ2_M are for when a model does not fit any other way. llama.cpp's quantize tool lists bits per weight for them but no perplexity figure, so test them on your own prompts before trusting them.

Speed depends on your hardware. bartowski's card recommends i-quants below Q4 on Nvidia (cuBLAS) and AMD (rocBLAS), and says they run slower than k-quants on a CPU. When IQ4_XS landed in February 2024, its author measured it slower than Q4_0 on Apple's Metal (53.9 against 63.1 tokens per second on an M2 Max GPU) and a little faster than Q4_0 on CUDA (133.7 against 128.8 on an RTX 4080). If you run on a Mac, time both before you settle.

To make a quant yourself, llama.cpp's README converts the model to a 16-bit GGUF first and then runs llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M. Adding --imatrix with an importance-matrix file is how bartowski made every quant in the Qwen3 32B repo.

Step 3: Estimate the memory cost#

You can predict a quantized file's footprint with simple arithmetic. Multiply the parameter count by the bits per weight, then divide by eight to get bytes:

bytes = parameters x bits_per_weight / 8

Use the file's real bits per weight, not the number in its name. Llama 3.1 8B at Q4_K_M is 8.03 billion x 4.89 / 8, about 4.9 GB, which matches the 4.92 GB file. That is why the GB-per-billion column above is the quicker rule: about 0.6 GB per billion parameters at Q4_K_M, 0.71 at Q5_K_M, 0.82 at Q6_K and 1.06 at Q8_0. For a mixture-of-experts model, use the total parameter count, because the file stores every expert.

Watch the units. The sizes in this guide are decimal gigabytes (bytes divided by 1,000,000,000), as bartowski's tables and Ollama's tag list show them, and so does llama.cpp's README memory table (the 70B figure in Step 4), but its README speed table uses GiB (1,073,741,824 bytes). A Llama 3.1 8B Q4_K_M file is 4.92 GB in the Hugging Face listing, and that speed table lists Q4_K_M on Llama 3.1 8B at 4.58 GiB.

On top of the weights you need room for the context window (the KV cache). llama.cpp stores it in 16-bit by default, and its size is 2 x layers x KV heads x head size x 2 bytes for every token. From each model's config (for Llama 3.1 8B, the GGUF file's own header):

With the default 16-bit cache, the cache's size does not depend on the weight quant you pick. On Llama 3.1 8B, a 32K-token cache (4.29 GB) is bigger than the whole step from Q4_K_M to Q8_0 (3.62 GB).

Step 4: Pick a quant for your VRAM#

The table below applies our rule: the file plus an 8K-token KV cache, with at least 1 GB of the card left over for the runtime and your desktop. bartowski's Qwen3 32B card gives a similar rule of thumb: pick a file 1 to 2 GB smaller than your VRAM.

The table covers only the three models and the 11 quants sized above; bartowski's repos also hold slightly larger files, such as Q4_K_L and Q6_K_L, that it leaves out. On a 48 GB card our parameters-first rule points past them: llama.cpp's quantize README lists Llama 3.1 70B at 43.1 GB at Q4_K_M, which leaves under 5 GB for the KV cache and the runtime. Work out the cache for your context before you choose it over 32B at Q6_K.

The "what we would run" column is our call, and it follows one rule: spend memory on parameters first, then on bits. Our reasoning: above Q4_K_M, llama.cpp's table shows each step up in bits buying back less than 0.12 of perplexity, so we would rather spend the memory on a larger model. Below Q4_K_M the cost rises fast, so a 32B squeezed into 16 GB at Q2_K is an experiment, not a default.

Q4_K_M is also Ollama's default download for Llama 3.1: llama3.1:latest is the same file (digest 46e0c10c039e) as its llama3.1:8b-instruct-q4_K_M tag. bartowski's card labels Q4_K_M "default size for most use cases".

To run this for your own card, the Local-LLM GPU Checker takes your GPU or its VRAM and shows which open models fit at each quantization. If you are still choosing hardware, our guide to the best GPUs for running LLMs locally compares cards by VRAM. On a Mac, memory is shared with macOS and your apps, and the unified memory calculator works out what a model actually gets.

Step 5: Verify the choice in practice#

Numbers on paper are a starting estimate. Confirm the fit by loading the model and watching three things:

If the model loads but answers feel degraded, move up one quantization step before blaming the model itself. If it will not fit, move down a step or shrink the parameter count.

Where this breaks#

One mistake to avoid is chasing higher bit counts when the real problem is too few parameters.

Eight bits of a tiny model is still a tiny model. Decide on parameter count first, then choose the largest quantization that fits, up to Q6_K.

The second pitfall is forgetting the context window. A long prompt or a big retrieval payload inflates the KV cache, and a model that fit comfortably with short prompts can run out of memory once you push the context. On Qwen3 32B, going from 8K to 32K tokens adds 6.44 GB, more than the whole gap between Q4_K_M and Q5_K_M.

Finally, do not assume a bigger model shrugs off aggressive quantization. When llama.cpp's k-quants were introduced in June 2023, their author measured LLaMA models from 7B to 65B and found that the relative quantization error, measured by perplexity, did not fall as the models got bigger. The perplexity figures above come from an 8B model and are a guide, not a guarantee for every model.

Test on your actual workload rather than trusting a single rule of thumb.