ArticleaiDeep read
How Much VRAM You Actually Need to Run a Local LLM
BitByteCore AI DeskAug 5, 20267 min
VRAM is the hard constraint on running a local LLM. Here's the real math — parameters, precision, quantization, KV cache — what fits on 8GB, 24GB, 32GB, and unified-memory machines, plus where quality and speed actually break.
A deep read — the full picture, with the receipts.
Why VRAM Is the Hard Constraint#
When a language model runs inference, its weights must live in GPU memory to access at high bandwidth. System RAM, connected via PCIe, is orders of magnitude slower than the GPU's native memory bus. The moment a model spills into system RAM—called CPU offloading—token generation speed collapses from tens of tokens per second to single digits or worse.
Unlike system RAM, consumer GPU VRAM is fixed at manufacture: 8 GB, 12 GB, 16 GB, 24 GB. You can't add more. That ceiling is real, which means fitting your model within it is an engineering problem with hard trade-offs, not a suggestion.
The Math: Parameters, Precision, and Memory#
Model memory footprint is determined by parameter count and the numerical precision per parameter:
- FP32 (32-bit float): 4 bytes per parameter
- FP16 / BF16 (16-bit float): 2 bytes per parameter
- INT8 (8-bit integer): 1 byte per parameter
- INT4 / Q4 (4-bit quantized): ~0.5 bytes per parameter
A 7-billion-parameter model at FP16 occupies roughly 14 GB for weights alone. Add the KV cache (which grows with context length and batch size), activations, and framework overhead, and you're looking at 15–17 GB minimum for comfortable inference. The same 7B model quantized to Q4 drops to 4–5 GB—a difference that moves it from impractical to accessible on mainstream consumer hardware.
Model Footprint by Size and Quantization
7B (Q4)
- ~4–5 GB weights
- Total: 5–7 GB (with overhead)
- Fits: 8 GB GPU comfortably
7B (FP16)
- ~14 GB weights
- Total: 15–17 GB
- Fits: 24 GB GPU tight
13B (Q4)
- ~7–9 GB weights
- Total: 8–11 GB
- Fits: 12–16 GB GPU
Key point
Quantization is not optional for consumer hardware—it's a practical necessity. A 4-bit quantized 7B model delivers genuinely capable performance on 8 GB VRAM, making local LLMs viable for most personal use cases.
What Quantization Actually Costs#
Quantization compresses weights from 16-bit to 4-bit by rounding each value to fit a smaller representable set. This introduces approximation error. For general conversation, summarization, and code completion, Q4 and Q5 quantization on well-trained models is largely indistinguishable from FP16 in practice—benchmarks show perplexity degradation, but perplexity doesn't always translate to noticeable quality loss.
Quantization hurts more noticeably on complex multi-step reasoning, precise mathematical computation, and tasks where the model is already near its capability edge. Very small models (under 3B parameters) also suffer more, since information density per parameter is already high.
Higher-quality quantization formats—GGUF with Q5_K_M or Q6_K, GPTQ, AWQ—recover much lost quality compared to naive INT4 rounding. If you're committing to a quantized model for serious work, pick a well-regarded quantization method.
Practical GPU Tiers and What You Can Run#
4–6 GB VRAM
Small, aggressively quantized models (1B–3B parameters at Q4). Usable for autocomplete, simple Q&A, lightweight coding assistance, but lacking reasoning depth for complex tasks. You'll hit the ceiling regularly.
8 GB VRAM (mainstream consumer)
A 7B model at Q4 fits comfortably with room for moderate context. This is the practical entry point for local LLM use right now—a well-quantized 7B (Mistral, LLaMA, Qwen, Phi) gives genuinely capable performance for writing, summarization, coding, conversation.
12 GB VRAM
Breathing room for 7B models at Q5/Q6, or 13B models at Q4. Better output quality, longer comfortable context windows. Noticeably better experience than 8 GB for quality-sensitive work.
16–24 GB VRAM (prosumer/enthusiast)
13B models at Q5/Q6, 34B models at Q4, mixture-of-experts models. Output quality becomes genuinely hard to distinguish from hosted API output on many tasks.
48 GB+ (workstation/multi-GPU)
70B models at Q4–Q5, or smaller models at full FP16. Serious development work, fine-tuning, multi-user inference. Requires runtimes supporting tensor parallelism.
Context Length: The Hidden Memory Multiplier#
The KV cache—which stores attention state for previous tokens—scales with both model depth and active context length. A 7B model at 2K tokens uses far less VRAM than the same model at 32K tokens. Many runtimes let you cap context length at inference time; if you're hitting memory pressure, reduce context before downsizing the model. You often lose less capability that way.
This matters more now because many modern models ship with large default context windows—128K tokens or more. Be intentional about what you enable at runtime.
CPU Offloading: A Bridge, Not a Strategy#
Runtimes like llama.cpp let you load part of a model's layers onto GPU and offload the rest to system RAM. This is useful for squeezing a bit more capacity—if you can fit 28 out of 32 layers on GPU, the remaining 4 on CPU barely hurt. But if you're fitting half the model on GPU and half on CPU, the speed penalty becomes frustrating for interactive use.
Use CPU offloading to bridge small gaps, not as a primary strategy. Also note that system RAM capacity matters more than people realize for hybrid setups—a machine with 16 GB system RAM is more constrained than one with 64 GB when offloading heavily.
Note
More VRAM beats a faster GPU clock for LLM inference. When choosing between GPU options, weight memory capacity heavily—throughput is bottlenecked by memory bandwidth, not compute.
The Bottom Line#
Local LLM deployment isn't magic—it's arithmetic. Model footprint scales linearly with parameter count and precision. Quantization (Q4–Q6) is the primary lever to fit large models into consumer hardware; quality loss is real but manageable with good formats. Context length multiplies memory use, so cap it deliberately. CPU offloading works for small gaps but degrades throughput significantly beyond that. Once you understand what drives VRAM consumption, the decisions become clear and the trade-offs stop being mysterious.
Sources
- Hugging Face — GGUF quantization typeshuggingface.co
- Hugging Face Transformers — KV cache strategieshuggingface.co
- llama.cpp — quantize tool and quantization typesgithub.com
- Kwon et al. — Efficient Memory Management for LLM Serving with PagedAttention / vLLM (arXiv)arxiv.org
- Dettmers et al. — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (arXiv)arxiv.org
Discussion