How to run a local LLM on your own machine with Ollama

A local LLM is a language model that runs entirely on your own hardware — no cloud account, no API key, and nothing leaving the machine. Ollama makes that straightforward: it downloads models, manages memory, and exposes them through both a terminal chat and a local HTTP endpoint. By the end of this guide you'll have one answering prompts on your own computer.

You need a machine with at least 8 GB of RAM (16 GB is more comfortable), a few gigabytes of free disk space, and an internet connection for the one-time model download. A discrete GPU helps but is not required. Ollama is built on the llama.cpp inference engine and auto-detects acceleration — Metal on Apple Silicon, CUDA on a recent NVIDIA card — and falls back to plain CPU when there's no GPU to use.

Step 1: Install Ollama#

Ollama ships installers for macOS, Windows, and Linux. On macOS and Windows you download the app and run it. On Linux the one-line script is the fastest path:

curl -fsSL https://ollama.com/install.sh | sh

When it finishes, confirm the binary is on your PATH:

ollama --version

If that prints a version string, the background service is already running and listening on localhost.

Step 2: Pull and run a model#

Models live in a registry, much like container images: you pull one once, the weights download to disk, and every later run reads from that local cache. Start with a small, current instruction-tuned model so the first download stays modest:

ollama run gemma3

The first invocation downloads the weights, then drops you into an interactive prompt. Type a question and the reply is generated locally, on your hardware. Wrap multi-line input in triple quotes ("""), and type /bye to exit. The weights stay cached, so the next ollama run starts in seconds.

Pick a model whose size fits your RAM. A rough rule: a 4-bit quantized model needs roughly its parameter count in gigabytes of memory once you include runtime overhead — a 3B model wants about 3 to 4 GB free, an 8B model closer to 8 GB. The small-model field is crowded and genuinely good right now: Google's Gemma 3 runs from under 1B up to 27B, with the 4B a strong laptop default; Alibaba's Qwen3 spans 0.6B to 32B; Meta's Llama 3.2 ships 1B and 3B sizes that suit very light machines. If your machine swaps hard or the reply crawls out one token at a time, drop to a smaller size.

Step 3: List and manage what you have#

Over time you'll collect several models. A few commands keep the set tidy:

ollama list
ollama ps
ollama rm gemma3

list shows every model on disk with its size, so you can see what's eating space; ps shows which models are currently loaded into memory and how much they're using; rm deletes weights you no longer want. To download a model without starting a chat — say, to pre-fetch it for a script — use ollama pull <model> instead of ollama run.

Step 4: Call the local HTTP API#

The interactive prompt is convenient, but the real power is the local server Ollama runs on port 11434. Any program on your machine can hit it, with no external dependency and no network round-trip. A plain one-shot request looks like this:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma3",
  "prompt": "Explain a hash map in two sentences.",
  "stream": false
}'

The response is JSON with the generated text in a response field. Set "stream": true to get tokens as they're produced — one JSON object per token — which is what you want behind a chat UI. For multi-turn conversations, POST to /api/chat with a messages array instead of a single prompt, and Ollama tracks the exchange as a back-and-forth. And if you already have code written against OpenAI's SDK, point it at http://localhost:11434/v1 — Ollama exposes an OpenAI-compatible /v1/chat/completions endpoint, so most client libraries work by changing only the base URL. Because the endpoints speak a stable JSON shape, you can wire a local model into scripts, editor plugins, or a small web app without shipping a single API key.

Step 5: Keep a model warm#

Loading weights into memory takes a few seconds — longer for bigger models. If you're calling the API repeatedly, that cold start adds up on every request that finds the model unloaded. Send a request with just the model name to load it and pin it in memory:

curl http://localhost:11434/api/generate -d '{"model": "gemma3", "keep_alive": "30m"}'

The keep_alive field sets how long Ollama holds the model in RAM after the last request. A duration like "30m" fits an interactive session; -1 keeps it resident until you unload it; 0 unloads it immediately after the call, which frees memory between bursts. Run ollama ps to confirm what's currently loaded.

Where this breaks#

The most common failure is memory pressure. If a model plus its context is larger than your free RAM, the OS pages weights to disk and generation slows to a crawl — or the process is killed outright. Remember that the footprint isn't just the weights: the context window grows it as the conversation gets longer, because the model has to hold a running key/value cache for every token in play. Watch system memory and step down a size before you blame the tool.

The second trap is expecting a small local model to match a frontier cloud model. A 3B or 4B model is genuinely useful for drafting, summarizing, classification, and tool routing, but it hallucinates more and reasons less reliably than the largest hosted systems. Match the model to the job, and reach for a bigger local size — or a cloud model — when the task demands it.

Finally, a quantized model trades some quality for size. If answers feel noticeably worse than you expected, try a less aggressive quantization or a larger parameter count before concluding the model is bad. The next guide in this cluster covers exactly how to read those quantization labels.