The best GPUs for running large language models locally in 2026

The short answer: the best GPU for running LLMs locally in 2026 is the NVIDIA RTX 5090. Its 32 GB of GDDR7 is the most memory of any GeForce card in this guide, and its 1,792 GB/s of bandwidth is what sets how fast a model writes. If your models are 32B or smaller, a used 24 GB card (an RTX 3090 or an RX 7900 XTX) runs them too at Q4, so price one against current 5090 listings. For Llama 70B, no single consumer card in this guide is enough at normal quality: the Q4_K_M file alone is 42.5 GB, so you need two 24 GB cards, a 96 GB RTX PRO 6000 Blackwell, or a 128 GB unified-memory machine such as NVIDIA's DGX Spark.

The catch is price. NVIDIA launched the 5090 at $1,999; check live listings before you buy. Memory supply is tight enough that NVIDIA has raised a price of its own: in February 2026 it raised the DGX Spark's MSRP from $3,999 to $4,699, citing industry-wide memory supply constraints.

So this isn't a simple "buy the 5090" list. It's a map of what each tier can run, how fast, and what the maker charges where it publishes a price.

Quick picks:

• Best GPU for local LLMs overall: NVIDIA RTX 5090 (32 GB), launched at $1,999; check live listings
• Best GPU for Llama 70B: two used RTX 3090s (48 GB together) hold it at Q4; the RTX PRO 6000 Blackwell (96 GB) does it on one card
• Best for running several models at once: RTX PRO 6000 Blackwell (96 GB), or a 128 GB DGX Spark or Ryzen AI Max+ 395 box if speed matters less
• Best used 24 GB card on AMD: AMD Radeon RX 7900 XTX (24 GB at 960 GB/s)
• Best used 24 GB card on CUDA: NVIDIA RTX 4090 (24 GB at 1,008 GB/s)
• Best new AMD card for LLMs: AMD Radeon AI PRO R9700 (32 GB)
• Best budget entry: Intel Arc B580 (12 GB), or an RTX 5060 Ti 16 GB if you want CUDA
• Best big-VRAM workstation card: Intel Arc Pro B70 (32 GB) if capacity matters more than speed; NVIDIA RTX PRO 6000 Blackwell (96 GB) if you need 96 GB on one card
• Best all-in-one "AI in a box": NVIDIA DGX Spark or AMD Ryzen AI Max+ 395 (128 GB unified); Apple Mac Studio with M5 Ultra if you want the most unified memory here (up to 512 GB)

---

The memory shortage behind today's prices#

You cannot make a sensible GPU decision in 2026 without understanding why prices look the way they do. Memory is in short supply, and every card in this guide is built on it. NVIDIA said as much about its own hardware: when it raised the DGX Spark's MSRP from $3,999 to $4,699 in February 2026, it cited "industry wide memory supply constraints".

The practical consequences for a buyer:

• Check street prices yourself. The prices in this guide are what each maker publishes (its list price, or the launch price where the page shows none), checked in October 2026; check live listings before you buy.
• Used cards are priced by the market. The used 3090, 4090 and 7900 XTX route still works; compare current listings, not older guides.
• Timing matters. If you don't need a card this week, it is reasonable to wait and watch, but don't pay for capacity you won't use. Buy for the model sizes you actually run today, not the ones you might run someday.

---

What actually matters for local LLM inference#

Before the picks, the one principle that drives all of them: local inference is a memory problem, not a compute problem. The model has to fit in memory first. Once it fits, generation speed is governed mostly by memory bandwidth: how fast weights stream to the compute units. A card with more bandwidth but lower raw TFLOPS will usually write tokens faster than a card with more TFLOPS and less bandwidth, because each new token means reading the weights again. And a card with more memory but lower bandwidth (the Intel Arc Pro and unified-memory boxes below) can run bigger models but will feel slower doing it. Capacity decides what you can run; bandwidth decides how fast. Keep that split in mind for everything that follows.

One more wrinkle for 2026: mixture-of-experts (MoE) models change the math. A 120B MoE model only activates a fraction of its parameters per token, so bandwidth-limited hardware (unified-memory boxes especially) runs large MoE models far faster than a dense model of the same nominal size. That's why a Strix Halo box can be slow on a dense 70B (a speed ceiling of about 6 tokens per second at Q4, as the 70B section below shows) yet much quicker on a large MoE model, which reads far fewer weights per token.

---

What each card can actually run#

If you've already bought the card, the question isn't which GPU to buy: it's which model will actually load. That is almost entirely a memory question, so you can answer it from VRAM alone.

Estimates for DENSE models at Q4_K_M and 8K (typical chat) context: roughly params × 0.55 GB × 1.18 for the KV cache. "Fits" means it fits with 15% headroom; a stretch pick fills the card with none to spare.

Two things are worth knowing. The first is that going from 32GB to 48GB only just lets a 70B-class model in: by the estimates above, a 72B model at Q4 (~46.7 GB) fills 48GB with nothing to spare. For comfortable use, the extra memory buys a better quantization of a 32B model, or a much longer context. The second is that a 70B model does not fit on any single consumer card in this guide. At Q4 with room for context it needs roughly 45GB, which means two 24GB cards, a workstation card, or a unified-memory machine.

If you want this for your exact card, quantization and context length, our Local-LLM GPU / VRAM checker does the arithmetic for the GPUs, Macs and open models it lists.

Which GPU runs Llama 70B?#

Two numbers decide it: the size of the model file and the context cache that sits beside it. For Llama 3.3 70B, the common GGUF files are 42.5 GB at Q4_K_M, 34.3 GB at Q3_K_M, 27.5 GB at IQ3_XXS and 21.1 GB at IQ2_XS; at the high-quality end, Q6_K is 57.9 GB and Q8_0 is 75.0 GB. The context cache adds about 0.33 MB per token at 16-bit (80 layers × 8 KV heads × 128 dimensions, once for keys and once for values, at 2 bytes each), so a typical 8K chat context costs about 2.7 GB and the model's full 128K window about 43 GB.

Fitting is half the answer; speed is the other half. On a dense model every generated token reads every weight once, so tokens per second can never beat memory bandwidth divided by model size. For the 42.5 GB Q4_K_M file that ceiling is about 6 tokens per second on a DGX Spark (273 GB/s) or a Ryzen AI Max+ 395 (256 GB/s), about 22 on two RTX 3090s split by layers (936 GB/s each, but the two halves are read one after the other), about 28 on a Mac Studio M5 Ultra (1.2 TB/s) and about 42 on an RTX PRO 6000 (1,792 GB/s). Real speeds land below the ceiling, never above it. That is why two used 3090s, with a ceiling near 22, have far more headroom on a 70B than the DGX Spark or Ryzen boxes, which can never pass about 6, and why those boxes are better at holding a 70B than at chatting with one.

Running several LLMs at the same time#

Memory adds up: two models loaded together need both sets of weights plus each one's context cache, and nothing is shared between them. Ollama's own FAQ sets the rules. It keeps several models loaded only if each new model fits completely in VRAM; it allows up to three loaded models per GPU by default (OLLAMA_MAX_LOADED_MODELS); and when a new model doesn't fit, requests wait in a queue while idle models are unloaded. Parallel requests cost memory too: Ollama multiplies a model's context by the number of parallel requests, so four parallel chats at 2K context reserve as much as one chat at 8K.

In practice, using the estimates in the table above: a 24 GB card holds a 14B chat model (~9.6 GB) and a 9B model (~5.8 GB) side by side. A 32 GB card can't comfortably hold a 32B model (~21.3 GB) plus a 14B one (~9.6 GB), because that leaves almost nothing for context. For two or three mid-size models at once, the simplest answer is one big pool: a 96 GB RTX PRO 6000, or a 128 GB unified-memory box if you can accept that every loaded model shares one pool of bandwidth (273 GB/s on the DGX Spark) and slows down when two of them write at the same moment. A multi-GPU rig is the build-it-yourself alternative: run one model server per card and each model gets that card's full bandwidth to itself.

NVIDIA RTX 5090: Best GPU for local LLMs overall#

The 5090 has the most memory bandwidth of any consumer card in this guide, 1,792 GB/s against the RTX 4090's 1,008 GB/s, and bandwidth is what sets how fast a model writes. Its Blackwell architecture brings native FP4 precision, a first for a consumer card, and FP4 weights take half the memory of FP8 ones. With 32 GB of GDDR7, it holds a 32B model at Q4 with room for an 8K context (about 21.3 GB, by the VRAM table above). Two of them hold Llama 70B at Q5_K_M (50 GB).

The drawback is price. NVIDIA launched it at $1,999 in January 2025; check live listings and price it against those, not against any number in an article, this one included. That doesn't make it a bad card: it makes it a card to buy when speed is genuinely your priority and 32 GB covers your models.

Who it's for: Anyone running 7B–34B models daily who wants fast generation and will hold the card for years. If you want to try 70B on one card, it holds a 3-bit file (IQ3_XXS, 27.5 GB) with room for context, which no 24 GB card can.

When to pick something else: If your budget won't stretch to current 5090 listings, if you only run sub-13B models, or if your real goal is 70B+ (where a multi-GPU rig or a big-memory box holds models a single 32 GB card can't).

---

Used RTX 3090 (and multi-GPU rigs): Best for Llama 70B on consumer cards#

A strong answer for anyone serious about local inference is not one card: it's two, three, or four used RTX 3090s. Each carries 24 GB of GDDR6X at 936 GB/s, and NVLink/PCIe multi-GPU splits let you stack that memory. A 3–4 card 3090 rig gives you 72–96 GB at 936 GB/s per card, against the DGX Spark's 128 GB at 273 GB/s, so on large models that fit, its weights stream far faster. For 70B–120B models on consumer cards, this is the build-it-yourself path.

Price a full rig (cards, motherboard, power supply) against a DGX Spark's $4,699 list price before you build.

Who it's for: Tinkerers who want to run big models fast on consumer cards, are comfortable building a multi-GPU box, and don't mind the power and heat that come with it.

When to pick something else: If you want a quiet, self-contained, low-power box, a DGX Spark or Strix Halo mini-PC trades throughput for convenience. If you only run small models, one 24 GB card is plenty.

---

AMD Radeon RX 7900 XTX: Best used 24 GB card on AMD#

The 7900 XTX is the 24 GB AMD consumer card in this guide, and it matters more now that the RX 9070 XT, AMD's newer RDNA4 gaming card, ships with 16 GB. It carries 24 GB of GDDR6 at 960 GB/s: matching the RTX 4090 on capacity, and close enough behind its 1,008 GB/s that bandwidth is not what decides between them. AMD's current compatibility matrix: ROCm 10.1.0, covering both Linux and Windows, names this exact card, Radeon RX 7900 XTX (gfx1100), among its supported GPUs, and AMD publishes its own Ollama-on-ROCm tutorials. Being listed by model rather than by series is the part that matters: AMD supports the cards on that list by name.

On price: listings, new or used, vary with condition and seller, so compare current ones.

If you want a newer architecture and don't need the full 24 GB, the RX 9070 XT (RDNA4, 16 GB) is named in AMD's ROCm 10.1.0 matrix and is worth a look: just accept that 16 GB caps you well below the 7900 XTX on model size.

If you want 32 GB on a new AMD card, the Radeon AI PRO R9700 is the one to look at. It's AMD's RDNA4 workstation card: 32 GB of GDDR6 on a 256-bit bus at 640 GB/s, 64 compute units and a 300 W board power, and ROCm 10.1.0 names it as Radeon AI PRO R9700 (gfx1201) on Linux and Windows. That is RTX 5090 capacity at about a third of the bandwidth, so it runs 32B models at higher-quality quantization and Llama 70B at 3-bit, just slower. AMD's product page lists no price, so check partner listings.

Who it's for: Enthusiasts who want 24 GB on AMD for 13B–27B models at Q4 with room for context, or a 32B at Q4 with almost none.

When to pick something else: If you're on Windows and won't troubleshoot driver quirks, pick NVIDIA. For 70B+, no 24 GB card holds it at Q4.

---

NVIDIA RTX 4090: Best used 24 GB card on CUDA#

The 4090 isn't new, but it's still a serious option. With 24 GB of GDDR6X and 1,008 GB/s of bandwidth, it runs models up to 27B at Q4 with room to spare, and a 32B at Q4 with almost none (about 21.3 GB, by the VRAM table above).

The 2026 caveat: price it against current 5090 listings. If the gap is small, the 5090's 1,792 GB/s and FP4 support make it the better long-term buy.

Who it's for: Anyone who finds a genuinely strong used deal, shops where 5090 supply is thin, or uses frameworks that assume an NVIDIA card.

When to pick something else: At near-equal pricing, buy the 5090. If you want more total VRAM, used 3090s in a multi-GPU rig add 24 GB per card.

---

Intel Arc B580: Best budget entry#

The B580 is for running small local models on a small budget. Intel's recommended customer price is $249 (check current listings), and it brings 12 GB of GDDR6 at 456 GB/s. That holds 7B models and 13B models at Q4. The B580 launched in Q4 2024. Intel archived its IPEX-LLM library on GitHub in January 2026, so check that the framework you plan to use supports Arc before you buy.

Who it's for: Genuinely budget-constrained buyers, or anyone who wants to experiment with local inference before committing more, primarily on 7B models for summarization, coding assist, or chat.

When to pick something else: If you can stretch to a used 7900 XTX, you double your headroom. If you want CUDA at this tier, the RTX 5060 Ti 16 GB (GDDR7 on a 128-bit bus) is the NVIDIA alternative: the extra 4 GB moves you from 14B-class models up to Mistral Small 3 24B at Q4, just.

---

Intel Arc Pro B70 (and the B60/B50): Best big-VRAM card if capacity matters more than speed#

Released in early 2026, the Arc Pro B70 is Intel's big-Battlemage workstation card: 32 GB of GDDR6 at 608 GB/s across a 256-bit bus, with 32 Xe2 cores, 367 INT8 TOPS and a 230 W board power. Intel's specification page lists no recommended customer price, and board partners set their own, so treat any figure you see as a listing rather than a list price. Bandwidth is much lower than a 5090's, so it's slower, but 32 GB runs 32B models at higher-quality quantization and Llama 70B at 3-bit, which 24 GB cards can't hold.

Below it, Intel's workstation tier steps down to the Arc Pro B60 (24 GB GDDR6, 456 GB/s, 197 INT8 TOPS, 200 W) and the Arc Pro B50 (16 GB, 224 GB/s, 170 INT8 TOPS, and only 70 W). Intel's specification pages list no recommended customer price for either, and board partners set their own, so treat any figure you see as a listing rather than a list price. What Intel does document is its plan for scaling the tier: Project Battlematrix workstations with up to eight B60s, for up to 192 GB of combined video memory. For labs that care about VRAM over raw speed, these are worth a serious look.

Who it's for: Workstation and lab builders who need capacity headroom for larger models and can live with slower generation.

When to pick something else: If speed matters more than model size, pay for the 5090. If you need 70B+ reliably with budget to spare, jump to the PRO 6000 Blackwell.

---

NVIDIA RTX PRO 6000 Blackwell: Best single card for 70B+#

This is not a consumer product. The RTX PRO 6000 Blackwell is a professional workstation GPU: 96 GB of GDDR7 with ECC, 1,792 GB/s of bandwidth and a 600 W board power. NVIDIA calls it "the most powerful desktop GPU ever created", and its 96 GB, three times a 5090's 32 GB, runs a 70B model at 8-bit (the Q8_0 file is 75 GB) or a 120B-class MoE model on one card. Not at 16-bit, though: full-precision 70B weights are about 140 GB, more than this card holds.

On price, NVIDIA's product page lists none and sends buyers to its marketplace and partners, so get a current reseller quote before you plan around it. That's the GPU itself; a matched workstation platform adds more.

Who it's for: Researchers, labs, and teams running 70B+ models for production inference, fine-tuning, or multi-model workloads on a single card. Not a hobbyist purchase.

When to pick something else: Any scenario where budget is finite and your targets stay under ~34B: a 5090, or a multi-GPU consumer rig, covers it.

---

The "AI in a box" tier: DGX Spark, Ryzen AI Max+ 395, and Apple Silicon#

There's a whole category now that sidesteps the GPU question: single boxes with large unified memory shared between CPU and GPU. They trade bandwidth (and therefore speed) for capacity, low power, and simplicity. They suit large MoE models far better than dense 70B decode. Three main paths:

NVIDIA DGX Spark (GB10 Grace Blackwell)#

NVIDIA's desktop AI system. It pairs a Grace Blackwell GB10 with 128 GB of coherent unified LPDDR5X and up to ~1 PFLOP of sparse FP4 compute, and NVIDIA raised the Founders Edition MSRP from $3,999 to $4,699 in February 2026, citing industry-wide memory supply constraints. NVIDIA rates one unit for models up to 200B parameters and two linked units (256 GB) for up to 400B. NVIDIA's page also lists a 64 GB version, available only through participating OEM partners. A single unit runs a 120B-class MoE model comfortably, and MoE models write far faster on it than dense ones, because each token reads only the active experts.

The hard limit is 273 GB/s of memory bandwidth. That suits MoE models, which read only their active experts, but bottlenecks dense 70B+ decode, and it's why a multi-3090 rig, at 936 GB/s per card, has far more headroom on a dense model. You're paying for a quiet, compact, low-power, fully supported CUDA box, not for peak speed.

Who it's for: People who want a self-contained NVIDIA/CUDA inference machine with lots of memory and none of the multi-GPU assembly, and who mostly run MoE or mid-size models.

Trade-off in one line: Capacity and convenience, capped by 273 GB/s: if throughput is the goal, look at a used-3090 rig (936 GB/s per card).

RTX Spark laptops: NVIDIA's RTX Spark laptops went on sale on October 16, 2026, from ASUS, Dell, HP, Lenovo, Microsoft and MSI. NVIDIA's announcement lists them starting at $2,599, with pricing that varies by region. Each pairs an NVIDIA Blackwell RTX GPU and a Grace CPU on one chip, with up to 128 GB of unified memory and up to 1 petaflop of FP4 AI performance. NVIDIA's announcement gives no memory bandwidth figure, so check a model's spec sheet for it before you buy one for LLMs.

AMD Ryzen AI Max+ 395 (Strix Halo)#

Note the exact name: it's Ryzen AI Max+ 395, with the "+" on "Max." It's an APU, and AMD specifies it as 16 Zen 5 cores, up to 40 RDNA 3.5 compute units (not RDNA4), an XDNA 2 NPU at up to 50 TOPS, and up to 128 GB of unified memory with up to 96 GB available to the graphics side. AMD lists 256-bit LPDDR5x-8000 memory, which works out to 256 GB/s: the DGX Spark's neighbourhood, not a discrete card's, and real throughput lands under that theoretical figure.

Its strength is MoE models, which read only their active experts for each token and so write far faster here than a dense model of the same size. Dense 70B is where it struggles: at 256 GB/s the ceiling is about 6 tokens per second at Q4 and about 3 at Q8, slow enough to change how you use it. Because the iGPU is RDNA 3.5, it does not get RDNA4-specific ROCm features, though RDNA3-class support and Ollama do apply.

AMD lists this part by name in the ROCm 10.1.0 matrix as Ryzen AI Max+ 395 (Radeon 8060S) (gfx1151), on Windows and, for Linux, on Ubuntu, so AMD documents the software support for this exact chip.

On October 12, 2026, AMD introduced a new Ryzen AI Halo developer platform built on the Ryzen AI Max+ PRO 495: up to 16 Zen 5 cores and 40 RDNA 3.5 compute units, like the 395, but with up to 192 GB of LPDDR5x unified memory, of which up to 160 GB can be given to the GPU. AMD rates it for 300-billion-parameter and larger models at 4-bit, and the ROCm 10.1.0 matrix already lists it as Ryzen AI Max+ PRO 495 (Radeon 8065S) (gfx1151). AMD's post gives no bandwidth figure and no price, so the speed figures in this guide are for the 395.

Who it's for: Laptop and mini-PC builders (Minisforum, ASUS, and similar) who want big models without a discrete GPU, wall power, or a tower, and who lean on MoE models.

Trade-offs: LPDDR5X bandwidth caps speed; dense 70B is slow; the memory is shared with the system; and you're locked to whatever platform the APU ships in.

Apple Silicon (Mac Studio M5 Ultra / M5 Max)#

The other major unified-memory path. A Mac Studio with M5 Ultra, on sale since September 22, 2026, goes up to 512 GB of unified memory, more than anything else here, at 1.2 TB/s, more than four times the DGX Spark's 273 GB/s. That makes it the strongest of the three on dense-model speed at capacity. Apple lists it from $5,499 (and lists the M5 Max model from $2,499, with up to 128 GB at up to 614 GB/s) and says the 512 GB configuration arrives in late October. Apple's MLX framework and llama.cpp's Metal backend both run on it. The catch is the platform: no CUDA, so some inference frameworks and tooling still assume NVIDIA, and the high-memory configurations cost more than the starting price.

Who it's for: People already in the Apple ecosystem who want the most unified memory in this guide and are comfortable outside CUDA.

---

Comparison table#

Prices are the maker's list price where the maker publishes one, checked in October 2026. Check live listings before buying.

---

How to choose#

1. Will the model fit? Start here. Look up your target model size and the quantization you'll accept. A 70B model at Q4 needs roughly 45 GB (a 42.5 GB file plus context); a 13B at Q8 needs roughly 13 GB. If the card can't hold it, nothing else matters. For 70B+ you're choosing between one big-VRAM workstation card, a multi-GPU rig, or a unified-memory box.

2. Bandwidth is your speed dial. Once it fits, tokens-per-second is mostly memory bandwidth. This is why a 5090 (1,792 GB/s) out-writes a 4090 (1,008 GB/s) on models both can hold, and why the 128 GB unified-memory boxes feel slow on dense 70B despite the huge capacity: LPDDR5X bandwidth in the DGX Spark's 273 GB/s range can't stream a dense model's weights fast enough.

3. MoE vs. dense changes the answer. If you mainly run large MoE models, the unified-memory boxes (DGX Spark, Strix Halo, Mac Studio) do far better than their bandwidth alone suggests, because each token reads only the active experts. If you run dense 70B, prioritize bandwidth: a multi-GPU consumer rig or a PRO 6000.

4. Ecosystem friction is real on AMD and Intel. On AMD, ROCm 10.1.0 supports the cards AMD lists, on Linux and Windows both; check that your framework supports your card too, because an unlisted card is a different experience from a listed one. On Intel, check that your framework supports Arc: Intel archived IPEX-LLM, its own LLM library, on GitHub in January 2026. If you just want to run Ollama and never think about drivers, an NVIDIA card spares you these checks.

5. Mind the market, and consider waiting. Memory supply is tight: NVIDIA raised the DGX Spark's price in February 2026 for that reason. If your workload fits a cheaper card, that is a reason not to overbuy VRAM you won't use. If you don't need the card this week and your needs aren't urgent, it's reasonable to watch prices before committing.

---

The call#

If speed is the priority and 32 GB covers your models, the RTX 5090 has the most bandwidth of any consumer card here (1,792 GB/s): just go in knowing NVIDIA launched it at $1,999, and check live listings first. If your real goal is big models and you're willing to build, the honest 2026 answer is a multi-GPU used-3090 rig: each card streams its weights at 936 GB/s, against 273 GB/s on a DGX Spark and 256 GB/s on a Ryzen AI Max+ 395. Want big models without building anything? A DGX Spark or a Ryzen AI Max+ 395 mini-PC (or a high-memory Mac Studio) trades speed for a quiet, self-contained box. And if you're just experimenting or running 13B-class models, a used RX 7900 XTX gives you 24 GB at 960 GB/s, and AMD's ROCm 10.1.0 matrix names it outright.

Whatever you pick: check live prices first, and buy for the models you run today, not the ones you might run later.