Ask about memory before compute: a model has to fit to run at all, and bandwidth sets the speed. CPU, discrete GPU, or unified memory. A short buying guide, plus an interactive picker that points you to the right path.
A practical guide to getting it right.
When people ask what hardware they need to run models locally, they usually ask about compute speed. That is the wrong first question. The binding constraints are memory capacity and memory bandwidth. A model has to fit in memory to run at all, and how fast that memory feeds the processor largely sets how fast it generates. Compute matters, but it is rarely what stops you.
Why memory comes first#
Two numbers decide most of the outcome. Capacity is whether the model fits: if the weights plus working state exceed available memory, the model does not run, or it spills to slower storage and crawls. Bandwidth is how fast memory feeds the processor: generating output reads through the model's weights repeatedly, so the speed of that memory path is usually the ceiling on tokens per second, more than raw compute. Keep both in mind, because each path trades them differently.
The three paths#
CPU and system memory is the universal fallback: cheap to expand, so capacity is rarely the problem, but ordinary system memory has far less bandwidth than graphics memory, so generation is slow. A discrete GPU pairs strong compute with high-bandwidth graphics memory, which is why it is fast, but its dedicated pool is usually small, which caps model size. Unified memory shares one large pool between CPU and GPU, so the accelerator reaches far more memory than a typical discrete GPU, which is ideal for fitting bigger models, at somewhat lower bandwidth than a top discrete card. Match the path to whether your real limit is model size, speed, or budget.
Which path fits you?#
What matters most to you?
A big-VRAM RTX card in a desktop tower. The fastest path for CUDA tooling; loud and power-hungry, and VRAM caps model size.
See the full pick →A large shared memory pool runs bigger models than a similarly priced GPU, quietly and at low power.
See the full pick →A small box that sips power and runs mid-size models. The least-fuss option for a desk.
See the full pick →The cheapest route to real VRAM and CUDA, if you are willing to build it yourself.
See the full pick →Rent a GPU when you need one. No upfront cost or noise; you pay per hour, and your data leaves your machine.
See the full pick →The right path depends on model size, budget, and whether your tooling needs CUDA. See the linked guides for specific picks and prices.
See the specific picks
Our current guides to the best GPUs, mini-PCs, home-server builds, and cloud providers for running local AI.
Read more →Do I need a GPU to run AI models locally?
Not necessarily. A CPU with plenty of system memory can run small models and batch jobs, just slowly, because system memory has low bandwidth. For interactive use at larger sizes you want either a discrete GPU or a unified-memory machine, whichever fits your model size and budget.
Is unified memory better than a discrete GPU for local AI?
For fitting big models, often yes: unified memory gives the accelerator one large shared pool, so it can hold models a small-VRAM GPU cannot. A top discrete GPU still wins on raw bandwidth and speed, and on CUDA-only tooling. Choose by whether your limit is model size or speed.
How much VRAM or memory do I actually need?
Enough to hold the model you want to run, plus overhead, ideally with headroom. A quantized mid-size model needs roughly its file size in fast memory to run comfortably. Buy more memory than you think you need, since you usually cannot add it later.
Sources
- Apple, AI and machine learning for developersdeveloper.apple.com
- NVIDIA, CUDA platform for accelerated computingdeveloper.nvidia.com
- Hugging Face, GGUF quantization typeshuggingface.co
Discussion