Set up GPU drivers and the toolkit for local AI work

Most "GPU not detected" errors are not hardware faults. They are a version disagreement between the layers that have to line up before a framework can touch the card: the kernel driver, the CUDA runtime your framework was built against, and the framework build itself. The part people miss in 2026 is that you usually do not install that middle layer by hand. The PyTorch or JAX wheel you pip install ships its own CUDA runtime inside it. So the real question is narrower than it looks: is your driver new enough for the runtime your wheel carries, and did you actually install the GPU build instead of the CPU one. Get that to line up once and the rest of local AI work stops fighting you. This walks the install in the order that avoids the usual breakage.

One thing to settle first: if you only want to run models, you may never touch a compiler. Ollama, LM Studio, and prebuilt llama.cpp bundle their own runtime and ask only for a recent driver. The manual toolkit dance below matters when you use a framework directly, or compile custom kernels (quantization, attention, custom ops), which is exactly where the version traps live.

Prerequisites#

• A GPU the vendor still supports, with enough memory for your target model. Check the model card before you start. Current local families like Qwen3, Gemma 3, Llama, and DeepSeek publish their memory footprint. A 24 GB card (RTX 3090 or 4090) comfortably runs many quantized 20-30B models; 32 GB (RTX 5090) buys headroom; Apple Silicon shares one pool of unified memory between CPU and GPU, so the ceiling is your total RAM rather than a separate VRAM figure. Weights that do not fit will not run no matter how clean the driver is.
• Admin rights on the machine, and a terminal you are comfortable in.
• A clean Python environment manager (venv or conda). Never install ML packages into the system Python.
• One fact written down before you begin: which framework build you need and the CUDA version it targets, which usually shows up as a wheel tag (something like cu124 or cu128). This single fact prevents most failures.

Step 1: Check what is already there#

Before installing anything, see the current state. A half-installed older driver is the most common source of conflicts.

# Vendor GPU query tool reports driver + detected devices
nvidia-smi
# AMD: rocm-smi   |   Apple Silicon: no query tool, the GPU is always present to Metal

Read the header carefully. The CUDA Version printed in the top right is the highest CUDA the driver can support, not a toolkit you have installed. People conflate those constantly. If it reads, say, 12.4, this driver can run CUDA builds up to that version. A small overshoot inside the same major version is usually fine: thanks to CUDA's minor-version compatibility (CUDA 11 onward), a wheel built for a slightly newer minor release, say a cu128 build on a driver that tops out at 12.4, will typically still initialize. What reliably fails is jumping a whole major version, like a CUDA 13 wheel on a driver capped at 12.x. If the command is missing, you have no working driver yet. Either way, write the number down.

Step 2: Install or update the kernel driver#

Use the vendor package channel or your distribution's repository, not a random binary. On Linux, prefer the packaged driver so kernel updates do not silently break it. On Windows, install the vendor package; if you run WSL2, install the driver on the Windows side only. Apple Silicon has no driver to install, since Metal ships with macOS. Reboot after installing so the kernel module actually loads.

# Re-run the query after reboot. You want a version and a listed device.
nvidia-smi

If the device still does not appear, stop here. A toolkit installed on top of a broken driver will not help.

Step 3: Install a system toolkit only if you compile#

For running models, you usually do not install one at all. The framework wheel bundles the CUDA runtime it needs, so all you require is a driver new enough to support it (the number from Step 1). In that case, skip to Step 4.

Install the full CUDA toolkit, the one that puts nvcc on your PATH, only when you compile GPU code yourself: a custom extension, flash-attention or bitsandbytes from source, or any package with no prebuilt wheel for your setup. When you do, match the toolkit's major and minor version to the runtime the framework targets, then confirm it is reachable.

# Only relevant if you are compiling. Confirm the toolkit compiler is on PATH.
nvcc --version

A mismatched system toolkit is harmless for pure inference, because the framework ignores it and uses its bundled runtime. It only bites when nvcc compiles against headers that disagree with the runtime the framework loads at import time.

Step 4: Install the framework build that matches#

Frameworks ship a separate build per CUDA version, plus a CPU-only build. Pick the CUDA build whose version your driver supports, generally at or below the number from Step 1. Grabbing the default CPU wheel by accident is a frequent and silent mistake, and pip will happily take it if you forget the index URL.

# Verify two things: which CUDA the wheel was built for, and that it sees the GPU.
python -c "import torch; print(torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

torch.version.cuda tells you which CUDA the wheel was compiled against; None means you installed the CPU build and nothing else you do will find the GPU. A CUDA version plus True plus your device name means the driver and the runtime agree. That is the goal.

Step 5: Run a tiny real workload#

Allocate a tensor on the device and do one operation. If this runs without an out-of-memory or driver error, you are done.

python -c "import torch; x = torch.randn(1000, 1000, device='cuda'); print((x @ x).sum().item())"

On Apple Silicon, use device='mps'. On an AMD ROCm build, the 'cuda' device string still works, because the ROCm build maps it onto HIP under the hood.

Pitfalls#

• CPU wheel instead of GPU wheel. This is the number one reason is_available() returns false. Check torch.version.cuda; if it is None, you installed the CPU build. Reinstall from the correct index and the problem disappears.
• Driver older than the wheel's CUDA. A wheel built for a CUDA major version above your driver's advertised maximum (the top-right number in nvidia-smi) will not initialize, for example a CUDA 13 wheel on a driver that tops out at 12.x. A minor-version overshoot within the same major, like cu128 on a 12.4-max driver, usually still runs, thanks to CUDA's minor-version compatibility, so the cross-major gap is the one you actually have to avoid. Newer drivers are backward compatible and run older builds fine, so updating the driver is usually the clean fix; installing an older CUDA build of the framework is the fallback when you cannot touch the driver.
• Mixed install sources. A driver from one channel plus libraries from another often disagree. Pick one source and stay on it.
• The WSL2 driver trap. On Windows with WSL2, the GPU driver goes on Windows, not inside the Linux environment. Installing a Linux driver in WSL breaks the passthrough. The toolkit, if you need one, goes inside WSL; the driver never does.
• Skipping the reboot. The kernel module loads at boot. A driver that "installed fine" but is not detected usually just needs a restart.
• Polluting the base environment. Always install into a fresh virtual environment so a broken attempt is a rm -rf away, not a full OS reinstall.
• Ignoring VRAM limits. A correct stack still fails with out-of-memory if the weights do not fit. Quantization (4-bit, 8-bit) is what lets a large model land on a small card. A mixture-of-experts model such as Qwen3's 30B-A3B activates only a few billion parameters per token, which makes it fast to run, but all of its experts still have to be resident in memory, so its footprint is roughly that of a dense model of the same total parameter count, not smaller. MoE buys speed, not a smaller card; quantization is the lever that makes something fit. Check the arithmetic before blaming the install.
• Forgetting PATH. If nvcc is missing but installed, the toolkit's bin directory is not on your PATH. Add it and reopen the shell. This only matters if you are compiling, per Step 3.

Once the layers agree, treat that combination as load bearing. Pin the versions in your environment file, the framework wheel tag and the driver version, so the next machine reproduces it instead of rediscovering the same afternoon of debugging. Write down the exact driver, the wheel's CUDA tag, and, if you compiled anything, the toolkit version, because the next time something breaks the first useful question is always whether one of them moved. A reproducible record turns a vague guessing session into a quick diff against a known-good setup.