Local-LLM GPU Checker

It comes down to your VRAM. At 4-bit (Q4_K_M) and a typical 8K chat, budget about 0.65 GB per billion parameters. The largest model the checker calls comfortable is Gemma 2 9B on 8 GB, Qwen3 14B on 12 or 16 GB, Gemma 4 31B on 24 GB and Qwen3 32B on 32 GB. Llama 3.3 70B needs 45.4 GB at the same settings. Pick your card in the checker above for the verdict on every model it carries.

At 4-bit, models up to Gemma 2 9B (5.8 GB with an 8K chat) run with room to spare on an 8 GB card such as the RTX 5060. Gemma 3 12B comes to 7.8 GB, which the checker calls tight: it loads, with little room for a longer context. Nothing larger in the checker fits at 4-bit; the checker's Q3_K_M setting trims a model further, at some quality loss.

Not on one card at 4-bit. Llama 3.3 70B needs 45.4 GB at Q4_K_M with an 8K chat, and the RTX 5090 has 32 GB. The largest model in the checker it runs comfortably is Qwen3 32B, at 21.3 GB, which leaves room for a longer context. On 48 GB of memory the checker calls the same 70B tight.

They compare the model's estimated memory with your card's. Comfortable means the model needs no more than 85% of the VRAM, leaving the rest for longer prompts and the runtime. Tight means it fits with less spare than that, so a longer context can push it over. No means the estimate is larger than the card. The estimate is parameters times bytes per weight times a context factor: a rule of thumb, not a measurement.

Rule of thumb: multiply the model's parameter count (in billions) by the bytes per weight for your quantization (2 GB/B for FP16, 1.06 GB/B for 8-bit, 0.55 GB/B for 4-bit (GGUF Q4_K_M)) then add about 18% for the KV cache, activations and framework overhead. So an 8B model at 4-bit needs roughly 5.2 GB, and a 70B model at 4-bit about 45.4 GB. Those are the exact numbers the checker above computes, because this page and that tool read the same constants.

Not on a single card at usable quality. Llama 3.3 70B at 4-bit (Q4_K_M) is about 39 GB of weights and 45.4 GB with a typical chat context, so it needs a machine with more than that: typically two 24 GB cards (dual 3090/4090) or an Apple machine with 64 GB+ unified memory. On one 4090, a 32B model (Qwen2.5 32B or DeepSeek-R1 32B) at 4-bit comes to 20.8 GB, which the checker calls tight — it runs, with little room for longer context.

At 4-bit you comfortably fit 7B–14B models (Llama 3.1 8B, Qwen2.5 14B, Phi-4 14B, Gemma 3 12B): a 14B comes to 9.1 GB with room for longer context. A 24B (Mistral Small 3) is 15.6 GB, which is tight at short context. A 27B (Gemma 3) needs 17.5 GB and does not fit at all — that is the checker's verdict, not a hedge. Note the 4060 Ti 16GB has a narrow memory bus, so it fits the same models but generates noticeably slower than a 4080/5080.

Yes, and it's one of Apple Silicon's biggest advantages: the GPU shares the whole unified memory pool. But macOS reserves a slice for the system, and that reserve barely shrinks with the machine, so the share you keep for model weights falls as the Mac gets smaller: about 75% of a 128 GB machine, 69% of a 16 GB one, 63% of an 8 GB one. A 64 GB Mac gives ~48 GB usable, which the checker calls tight for a 70B at 4-bit — it runs, with nothing spare for a longer context. 128 GB gives ~96 GB usable, which is comfortable for the same model. You can raise the GPU memory cap with a sysctl tweak if you need more.

For local chat, 4-bit (Q4_K_M) is what almost everyone runs: it cuts VRAM 73% versus FP16 while losing only ~1.5–2 points on benchmarks. 8-bit is near-lossless (under ~1 point) but takes 1.9× the memory of 4-bit. FP16 is reference quality but rarely worth it locally; it's mostly for training/fine-tuning, not inference.

Beyond the fixed weight memory, the model stores a KV cache for every token in the conversation, and that grows with context length. A short 4K chat adds only ~10% overhead, but a full 128K-token context adds about 70% on top of the weights, which can push a model that 'fits' at 8K into 'won't fit' at 128K.