How to run a local LLM on your own machine with Ollama

The short answer: to run a local LLM with Ollama, install Ollama (one command on macOS, Windows or Linux, or the app from ollama.com/download), then run ollama run gemma4:e2b in a terminal. Ollama downloads the model once, opens a chat on your computer, and serves the same model to your own programs at http://localhost:11434. Local models need no account and no API key. To run Llama locally, ollama run llama3.2 fetches Meta's 3B Llama 3.2, a 2.0 GB download. A GPU makes it faster but is optional: Ollama supports NVIDIA cards from compute capability 5.0, recent AMD Radeon cards and Apple's M-series chips, and it runs on the CPU when there is no supported GPU.

1. Install Ollama for macOS, Windows or Linux.
2. Run ollama run gemma4:e2b (or ollama run llama3.2) to download a model and start chatting.
3. Type /bye to leave, then run ollama ps to see whether the model loaded on the GPU or the CPU.
4. Call the model from code at http://localhost:11434/api/chat, or point an OpenAI client at http://localhost:11434/v1.

A local LLM is a language model that runs on your own hardware, so your prompts and its answers are processed on your machine. Ollama handles the fiddly parts: it downloads models from its library, loads them onto your GPU or CPU, and exposes them through a terminal chat and a local HTTP API. Ollama's FAQ says it does not see your prompts or data when you run locally. Ollama now also offers cloud models, whose tags end in cloud, such as gemma4:cloud or gemma4:31b-cloud; those run on Ollama's servers, so for a fully local setup use tags without that ending. This guide was checked against Ollama's documentation on October 23, 2026, when the latest release on GitHub was v0.40.10, published October 23, 2026. If you are still choosing a runner, our comparison of Ollama, LM Studio, vLLM and other local LLM runners covers the alternatives.

What you need before you start#

• macOS: Sonoma (version 14) or newer. Apple M-series Macs get CPU and GPU support; Intel Macs run models on the CPU only.
• Windows: Windows 10 22H2 or newer, Home or Pro. With an NVIDIA card, driver 551.61 or newer; with an AMD Radeon card, a ROCm v7 (HIP 7) capable driver, or a Vulkan-capable Radeon driver.
• Linux: Ollama publishes packages for x86-64 (amd64) and ARM64. NVIDIA cards need driver 550 or newer; AMD cards need ROCm v7.
• Disk: Ollama's Windows guide asks for at least 4 GB for the install itself, and warns that models "can be tens to hundreds of GB in size". Small ones are far lighter: Ollama's library lists gemma3:1b at 806 MB and llama3.2 at 2.0 GB.
• Memory: for gemma4:e2b, Ollama's quickstart recommends 8 GB of available VRAM, or unified memory on a Mac. With less VRAM, Ollama can use system RAM, but responses may be slower. Our local LLM GPU checker shows which open models fit your card, and at which quantization.

Install Ollama on macOS, Windows or Linux#

Ollama's download page gives one install command per system. On macOS or Linux, in Terminal:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, in PowerShell:

irm https://ollama.com/install.ps1 | iex

If you prefer a normal installer, download Ollama.dmg on a Mac and drag Ollama into Applications, or run OllamaSetup.exe on Windows, which installs into your user account without administrator rights. On Windows the ollama command then works in cmd, PowerShell or any terminal, and Ollama keeps running in the background.

Check that the command line can reach the background server:

ollama -v

If it prints ollama version is followed by a number, you are ready. If it prints "Warning: could not connect to a running Ollama instance", open the Ollama app (macOS or Windows) or start the server on Linux with ollama serve (or sudo systemctl start ollama if it was installed as a service), then try again. Running ollama on its own opens an interactive menu that offers to run a model or connect Ollama to tools such as Claude Code, Codex or OpenCode.

Run your first local model#

Models come from Ollama's library, much like container images: you pull one once, the weights are saved to disk, and later runs load from that copy. Ollama's own quickstart starts with Google's Gemma 4 E2B (the "E" stands for "effective" parameters, Google's label for its small edge models):

ollama run gemma4:e2b

The first run downloads the model, then opens an interactive prompt. Type a question and the reply is generated on your machine. Wrap multi-line input in triple quotes ("""), and type /bye to leave.

The same command runs Meta's Llama models: ollama run llama3.2 pulls the 3B Llama 3.2, and ollama run llama3.2:1b the 1B version. Swap in any name from Ollama's library the same way.

Pick a model that fits your memory. These tags and sizes are as Ollama's library listed them on October 23, 2026; some tags list a range of sizes.

A rough rule for 4-bit models: plan on about one gigabyte of free memory per billion parameters. Hugging Face's GGUF documentation puts the common Q4_K format at 4.5 bits per weight, so the weights of an 8B model alone take about 4.5 GB (our arithmetic), and the context cache and runtime come on top. If your machine starts swapping, or replies crawl out a word at a time, drop to a smaller tag. Our What GPU do I need tool shows the smallest GPU or Mac for a given model and quantization, and How Much VRAM You Actually Need explains the sums.

Check whether Ollama is using your GPU#

Ask the model something, then run ollama ps. The PROCESSOR column shows where the model loaded: 100% GPU means entirely in GPU memory, 100% CPU means entirely in system memory, and a split such as 48%/52% CPU/GPU means part of it did not fit on the card. A CPU or split load is the first thing to check when a model feels slow.

What Ollama's hardware support page listed on October 23, 2026:

• NVIDIA: compute capability 5.0 or newer with driver 550 or newer; cards at compute capability 5.0 to 6.2 need driver 570 or newer. Ollama's table runs from the GeForce GTX 750 and GTX 750 Ti to the RTX 50 series. Look up your card on NVIDIA's CUDA GPU compute capability page, and run nvidia-smi to confirm the driver works.
• AMD on Linux: ROCm v7, for the Radeon RX 9000 and 7000 series, the RX 6800 and 6900 class, Radeon PRO and Radeon AI PRO cards, Ryzen AI chips such as the Ryzen AI Max+ 395, and Instinct accelerators.
• AMD on Windows: ROCm v7 for the Radeon RX 7600, 7600 XT, 7700 XT, 7800 XT, 7900 GRE, 7900 XT and 7900 XTX, and the Radeon PRO W7500 to W7900. Other Radeon cards can use Vulkan, which Ollama enables by default.
• Other GPUs, including Intel: Vulkan on Windows and Linux. Windows drivers usually include Vulkan; most Linux distributions need extra packages.
• Apple: M-series Macs run models on the GPU through Metal, Apple's graphics and compute API.

If your card needs drivers first, our guide to setting up GPU drivers and the toolkit for local AI walks through NVIDIA and AMD.

List, stop and remove models#

A few commands keep your collection tidy:

ollama list
ollama ps
ollama stop gemma4:e2b
ollama rm gemma4:e2b

ollama list (or ollama ls) lists the models on disk; ollama ps shows which are loaded in memory right now, and until when; ollama stop unloads a model without deleting it; ollama rm deletes its weights. To download a model without opening a chat, for example before a script needs it, use ollama pull gemma4:e2b.

Models are stored in ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux and C:\Users\%username%\.ollama\models on Windows. To keep them on a bigger drive, set the OLLAMA_MODELS environment variable to another folder and restart Ollama.

Call the local API on port 11434#

While Ollama runs, it serves an HTTP API at http://localhost:11434. By default it listens only on 127.0.0.1, so only programs on your own computer can reach it, and local requests need no API key. A one-shot request:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4:e2b",
  "prompt": "Explain a hash map in two sentences.",
  "stream": false
}'

The reply is one JSON object with the text in its response field. Streaming is the default: leave out "stream": false and the answer arrives as a series of JSON objects, one per line, as it is generated, which is what a chat interface wants. For a conversation, POST to /api/chat with a messages array; the answer is in message.content.

If your code already uses OpenAI's SDK, set its base URL to http://localhost:11434/v1. Ollama implements a subset of the OpenAI API there, including /v1/chat/completions, /v1/responses and /v1/embeddings; the client insists on an API key value, which Ollama ignores. There is also an Anthropic-compatible /v1/messages endpoint, and official Ollama libraries for Python and JavaScript.

Changing the listening address with the OLLAMA_HOST environment variable lets other computers reach the API, and with it anything else on that network. Read the security risks of running a local LLM before you do, and see Serve a local model as an API endpoint for a fuller setup.

Keep a model loaded between requests#

Ollama keeps a model in memory for 5 minutes after the last request, then unloads it, so the next call waits while the weights load again. To preload a model and choose how long it stays, send a request with only the model name and a keep_alive value:

curl http://localhost:11434/api/generate -d '{"model": "gemma4:e2b", "keep_alive": "30m"}'

keep_alive takes a duration such as "30m" or "24h", a number of seconds such as 3600, any negative number such as -1 to keep the model loaded, or 0 to unload it as soon as the response is done. The OLLAMA_KEEP_ALIVE environment variable sets the same thing for every model when the server starts, and a keep_alive in a request overrides it. From the terminal, ollama run gemma4:e2b "" preloads a model and ollama stop gemma4:e2b unloads it. Run ollama ps to see what is loaded.

Common problems and how to fix them#

It is slow. Run ollama ps. If PROCESSOR shows CPU, or a CPU/GPU split, the model does not fit in your GPU memory: pick a smaller tag or a lower-bit quantization. Our explainer on memory bandwidth and local LLM speed covers what decides speed once it fits.

It runs out of memory in long chats. The footprint is not just the weights: the context window needs a key/value cache that grows with the context length. Ollama picks the default context from your VRAM (4K tokens under 24 GiB, 32K from 24 to 48 GiB, 256K at 48 GiB or more), and its docs suggest at least 64,000 tokens for agents, web search and coding tools. Raise it with OLLAMA_CONTEXT_LENGTH or the slider in the app's settings only if you have the memory. Setting OLLAMA_KV_CACHE_TYPE=q8_0 uses about half the cache memory of the default f16, with what Ollama's FAQ calls "a very small loss in precision"; it needs Flash Attention, which Ollama turns on automatically where the hardware supports it.

The GPU is not used. Check the driver first (nvidia-smi on NVIDIA). On Linux, after a suspend and resume Ollama can lose an NVIDIA GPU and fall back to the CPU; Ollama's docs suggest reloading the driver with sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm. The server log says what Ollama found: ~/.ollama/logs/server.log on macOS, journalctl -u ollama on Linux, and server.log in %LOCALAPPDATA%\Ollama on Windows.

Answers are weaker than you hoped. Small models score lower than big ones: in the Gemma 4 benchmark table on Ollama's library page, E2B scores 60.0% on MMLU Pro against 85.2% for the 31B model. Try a larger tag or a less aggressive quantization before giving up on a model; our guide to choosing a quantization for a local LLM explains labels such as Q4_K_M and Q8_0.

A model tag ends in cloud. Tags such as gemma4:cloud and gemma4:31b-cloud run on Ollama's servers, not on your machine, and need you to sign in. To keep Ollama local only, set OLLAMA_NO_CLOUD=1, or "disable_ollama_cloud": true in ~/.ollama/server.json, and restart Ollama. That also turns off its web search.