Set up GPU drivers and the toolkit for local AI work

Most "GPU not detected" errors are not hardware faults. They are a version disagreement between the layers that have to line up before a framework can touch the card: the kernel driver, the CUDA runtime your framework was built against, and the framework build itself. The part people miss in 2026 is that you usually do not install that middle layer by hand. The PyTorch or JAX wheel you pip install ships its own CUDA runtime inside it.

So the real question is narrower than it looks: is your driver new enough for the runtime your wheel carries, and did you actually install the GPU build instead of the CPU one.

Get that to line up once and the rest of local AI work stops fighting you. This walks the install in the order that avoids the usual breakage.

One thing to settle first: if you only want to run models, you may never touch a compiler. Ollama, LM Studio, and prebuilt llama.cpp bundle their own runtime and ask only for a recent driver. The manual toolkit dance below matters when you use a framework directly, or compile custom kernels (quantization, attention, custom ops), which is exactly where the version traps live.

Prerequisites#

Step 1: Check what is already there#

Before installing anything, see the current state. A half-installed older driver is the most common source of conflicts.

# Vendor GPU query tool reports driver + detected devices
nvidia-smi
# AMD: rocm-smi   |   Apple Silicon: no query tool, the GPU is always present to Metal

Read the header carefully. The CUDA Version printed in the top right is the highest CUDA the driver can support, not a toolkit you have installed. People conflate those constantly. If it reads, say, 12.4, this driver can run CUDA builds up to that version. A small overshoot inside the same major version is usually fine: thanks to CUDA's minor-version compatibility (CUDA 11 onward), a wheel built for a slightly newer minor release, say a cu128 build on a driver that tops out at 12.4, will typically still initialize. What reliably fails is jumping a whole major version, like a CUDA 13 wheel on a driver capped at 12.x. If the command is missing, you have no working driver yet. Either way, write the number down.

Step 2: Install or update the kernel driver#

Use the vendor package channel or your distribution's repository, not a random binary. On Linux, prefer the packaged driver so kernel updates do not silently break it. On Windows, install the vendor package; if you run WSL2, install the driver on the Windows side only. Apple Silicon has no driver to install, since Metal ships with macOS. Reboot after installing so the kernel module actually loads.

# Re-run the query after reboot. You want a version and a listed device.
nvidia-smi

If the device still does not appear, stop here. A toolkit installed on top of a broken driver will not help.

Step 3: Install a system toolkit only if you compile#

For running models, you usually do not install one at all. The framework wheel bundles the CUDA runtime it needs, so all you require is a driver new enough to support it (the number from Step 1). In that case, skip to Step 4.

Install the full CUDA toolkit, the one that puts nvcc on your PATH, only when you compile GPU code yourself: a custom extension, flash-attention or bitsandbytes from source, or any package with no prebuilt wheel for your setup. When you do, match the toolkit's major and minor version to the runtime the framework targets, then confirm it is reachable.

# Only relevant if you are compiling. Confirm the toolkit compiler is on PATH.
nvcc --version

A mismatched system toolkit is harmless for pure inference, because the framework ignores it and uses its bundled runtime. It only bites when nvcc compiles against headers that disagree with the runtime the framework loads at import time.

Step 4: Install the framework build that matches#

Frameworks ship a separate build per CUDA version, plus a CPU-only build. Pick the CUDA build whose version your driver supports, generally at or below the number from Step 1. Grabbing the default CPU wheel by accident is a frequent and silent mistake, and pip will happily take it if you forget the index URL.

# Verify two things: which CUDA the wheel was built for, and that it sees the GPU.
python -c "import torch; print(torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

torch.version.cuda tells you which CUDA the wheel was compiled against; None means you installed the CPU build and nothing else you do will find the GPU. A CUDA version plus True plus your device name means the driver and the runtime agree. That is the goal.

Step 5: Run a tiny real workload#

Allocate a tensor on the device and do one operation. If this runs without an out-of-memory or driver error, you are done.

python -c "import torch; x = torch.randn(1000, 1000, device='cuda'); print((x @ x).sum().item())"

On Apple Silicon, use device='mps'. On an AMD ROCm build, the 'cuda' device string still works, because the ROCm build maps it onto HIP under the hood.

Pitfalls#

Once the layers agree, treat that combination as load bearing. Pin the versions in your environment file, the framework wheel tag and the driver version, so the next machine reproduces it instead of rediscovering the same afternoon of debugging. Write down the exact driver, the wheel's CUDA tag, and, if you compiled anything, the toolkit version, because the next time something breaks the first useful question is always whether one of them moved. A reproducible record turns a vague guessing session into a quick diff against a known-good setup.