Set up GPU drivers and the toolkit for local AI work

Short answer: to run local models in a runner such as Ollama, you only need a current GPU driver. Ollama's hardware page asks for NVIDIA driver 550 or newer (570 or newer for compute capability 5.0 to 6.2, which includes the GTX 10 series), ROCm v7 for AMD cards on Linux, and nothing extra on Apple Silicon, where it uses Metal. To use PyTorch or JAX directly, install the driver and then the framework's own CUDA build: those pip wheels bring the CUDA libraries with them, so the full CUDA Toolkit (the one with nvcc) is only needed when you compile GPU code. The number to check is the driver version. NVIDIA's release notes say CUDA 13.x builds need driver 580 or newer and CUDA 12.x builds need 525 or newer.

When a framework reports no GPU on a machine whose card works, check the software before the hardware. Three layers have to agree before a framework can touch the card: the kernel driver, the CUDA runtime your framework was built against, and the framework build itself. The PyTorch or JAX wheel you pip install carries that middle layer, so you rarely install it by hand.

So the real question is narrower than it looks: is your driver new enough for the CUDA build your framework carries, and did you install the GPU build instead of the CPU one?

Not sure your card has the memory for the model you want? The Local-LLM GPU checker shows which open models fit your VRAM, and What GPU do I need? works the other way, from model to card.

Which path you need#

If you are in the first row, steps 1 and 2 below are all you need, then see how to run a local LLM with Ollama.

Before you start#

• A GPU the current software still supports. NVIDIA: CUDA 13.0 removed support for Maxwell, Pascal and Volta, the generations before Turing, and JAX says its CUDA 13 wheels need compute capability (SM) 7.5 or newer. Ollama's table puts the GTX 10 series at compute capability 6.1, so those cards need CUDA 12.x builds; Ollama itself still supports compute capability 5.0 and up, but its hardware page says cards from 5.0 to 6.2 need driver 570 or newer. AMD: AMD's ROCm system requirements list the Radeon RX 9070 XT, 9070 GRE, 9070, 9060 XT, 9060 XT LP and 9060, and the RX 7900 XTX, 7900 XT, 7900 GRE, 7800 XT, 7700 XT and 7700, on Ubuntu 24.04.4, Ubuntu 22.04.5, RHEL 10.1 and RHEL 9.7, and say a GPU not on the list is not officially supported. Apple: PyTorch's GPU backend on a Mac (MPS) needs Apple silicon and macOS 14.0 or later.
• Enough memory for the model. Weights that do not fit will not run, however clean the driver is. NVIDIA lists 24 GB on the RTX 4090 and 32 GB on the RTX 5090; a Mac with Apple silicon shares one pool of memory between CPU and GPU. Check the fit with the GPU checker or our VRAM guide before you start.
• Admin rights on the machine, and a terminal you are comfortable in.
• A fresh Python environment (venv or conda). Never install ML packages into the system Python.
• One fact written down: which CUDA build of your framework you will install. On PyTorch's install page today (PyTorch 2.14.1) that is CUDA 12.6 or CUDA 13.2, shown in the command as cu126 or cu132.

1. Check what is already installed#

Before installing anything, see the current state. NVIDIA's CUDA Linux guide says to uninstall any previous installation that could conflict before you install.

# NVIDIA: driver version and detected GPUs
nvidia-smi
# AMD on Linux: AMD SMI replaces the older rocm-smi; rocminfo lists ROCm devices
amd-smi list
rocminfo
# Apple Silicon: nothing to query here; check from PyTorch in step 5

Read the nvidia-smi header carefully. The CUDA Version in the top right belongs to the driver, not to a toolkit you installed. NVIDIA's own compatibility guide shows driver 580.65.06 reporting CUDA Version: 13.0 while a CUDA 13.4 application runs on it, because from CUDA 11 onward a driver can run newer builds from the same major family (minor version compatibility). So the useful test is the driver number against this table:

NVIDIA lists two exceptions: code that relies on PTX compiled at run time, or on features that need a newer driver, wants a driver from the same CUDA minor release or later. If nvidia-smi is missing, you have no working driver yet. Either way, write the driver version down.

2. Install or update the GPU driver#

Install the driver on its own, from the vendor's repository or your distribution's packages, not a random download. The CUDA Toolkit will not do it for you any more: NVIDIA's release notes say the driver is no longer bundled with the toolkit on Windows from CUDA 13.1 and on Linux from CUDA 13.4.

• Linux, NVIDIA. NVIDIA's driver guide covers Ubuntu 22.04, 24.04 and 26.04 (and other distributions on their own pages). The network-repository route on Ubuntu is below; replace <distribution> with your release's code from NVIDIA's table: ubuntu2204, ubuntu2404 or ubuntu2604. Reboot when it finishes, as the guide's last step says.
• Windows. Install the current driver from NVIDIA's download page. Ollama for Windows asks for 551.61 or newer. If you use WSL2, this Windows driver is the only one you install (see the pitfalls below).
• Linux, AMD. Follow AMD's ROCm install guide for your distribution, then add yourself to the render and video groups and reboot, as AMD's guide says.
• Apple Silicon. Nothing to install. Metal ships with macOS.

# Ubuntu, NVIDIA's network repository (NVIDIA Driver Installation Guide)
sudo apt install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/<distribution>/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install nvidia-open
sudo reboot

# AMD on Linux, after installing ROCm (AMD's ROCm install guide)
sudo usermod -a -G render,video $LOGNAME
sudo reboot

After the reboot, run nvidia-smi again (or cat /proc/driver/nvidia/version, which NVIDIA's guide uses to show the loaded driver). You want a driver version and a listed GPU. If the GPU still does not appear, stop here: a toolkit or framework installed on top of a broken driver will not help.

3. Skip the CUDA Toolkit unless you compile#

For running models you usually do not need it. PyTorch's CUDA wheels bring the CUDA runtime libraries with them (on Linux, the metadata of PyTorch's cu132 wheel pulls in the CUDA 13.2.2 libraries from pip), and JAX's jax[cuda13] wheels do the same; the JAX team strongly recommends that route over a CUDA you installed yourself. All you need is a driver new enough for that build (the table in step 1). In that case, go to step 4.

Install the full CUDA Toolkit, the one that puts nvcc on your PATH, only when you compile GPU code yourself: a custom extension, a quantization or attention kernel built from source, or any package with no prebuilt wheel for your setup. NVIDIA's Linux guide recommends the cuda-toolkit package, and cuda-toolkit-X-Y if you want to stay on one release. Match its major and minor version to the CUDA build your framework uses, then make sure the compiler is on your PATH.

# Only if you compile. NVIDIA's guide for CUDA 13.4 adds this directory to PATH:
export PATH=/usr/local/cuda-13.4/bin${PATH:+:${PATH}}
nvcc --version

A system toolkit mostly sits unused by pip-installed frameworks, with one known trap: JAX's install guide warns that LD_LIBRARY_PATH can override the CUDA libraries it installed, so make sure that variable is not set when the wrong CUDA version shows up.

4. Install the framework build that matches#

Frameworks ship one build per CUDA version, plus a CPU-only build. Copy the exact command from the framework's install page instead of typing it from memory. On PyTorch's page today (stable 2.14.1), the choices are:

The index URL is what picks the GPU build. On Windows, PyTorch's CPU command is the plain pip3 install torch torchvision, so leaving the URL off there gets you a CPU-only build. PyTorch's page lists no ROCm build for Windows; AMD publishes its own Windows PyTorch for Radeon cards (with ROCm 7.2.1, per AMD's Radeon and Ryzen docs). If your NVIDIA card is older than Turing, CUDA 13 cannot target it, so look at the CUDA 12.x build and check PyTorch's release notes for your card.

For JAX on Linux, the JAX guide's command is pip install --upgrade "jax[cuda13]", with jax[cuda12] as the alternative. Its CUDA wheels are Linux only.

# Verify which build you got and whether it sees the GPU
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.version.hip, torch.cuda.is_available())"

If you installed from PyTorch's index URL, the part of torch.__version__ after the + names the build, exactly as in PyTorch's wheel index: +cu132, +cu126, +rocm7.14 or +cpu. A plain pip3 install torch from PyPI has no + tag (PyPI lists the release as plain 2.14.1), so there the last value is the test. A +cpu build, or False at the end, means nothing else you do will find the GPU; reinstall with the right index URL. On a Mac, False is expected here: use the MPS check in step 5. On a ROCm build, torch.version.hip is set (PyTorch's own HIP notes use it to tell the two builds apart). A CUDA or HIP version plus True means the driver and the runtime agree. That is the goal.

5. Run one real operation on the GPU#

Allocate a tensor on the device and do one operation. If this runs without an out-of-memory or driver error, you are done.

# NVIDIA, and AMD ROCm builds too
python -c "import torch; x = torch.randn(1000, 1000, device='cuda'); print((x @ x).sum().item())"
# Apple Silicon (Apple's check prints tensor([1.], device='mps:0'))
python -c "import torch; print(torch.backends.mps.is_available()); print(torch.ones(1, device='mps'))"

On an AMD ROCm build, the 'cuda' device string is the right one: PyTorch's HIP notes say ROCm reuses the torch.cuda interfaces and that 'rocm' or 'hip' are not valid device names.

If you only use Ollama, load a model and run ollama ps. Ollama's FAQ says the PROCESSOR column shows 100% GPU when the model sits entirely in GPU memory, 100% CPU when it does not use the GPU at all, and a split such as 48%/52% CPU/GPU when it only partly fits.

Why the GPU is still not detected#

Once the layers agree, treat that combination as load bearing. Write down the driver version, the framework's build tag (such as +cu132), and, if you compiled anything, the toolkit version, and pin them in your environment file. The next time something breaks, the first useful question is whether one of them moved, and a written record turns that into a quick comparison. If you are still choosing hardware, our GPU guide and what switching away from CUDA breaks cover the vendor side.