How to build a basic RAG pipeline for a local LLM

By the end of this walkthrough you will understand how a retrieval-augmented generation (RAG) pipeline lets a language model answer questions from your own documents instead of only what it absorbed during training. You need a local LLM runtime — an Ollama setup works well — an embedding model, and a small collection of text you want to query: notes, docs, a knowledge base, anything. No cloud services are required, and nothing you index leaves your machine.

RAG exists to solve one problem: a model only knows what it was trained on, and its context window is finite, so you cannot just paste your whole corpus in front of every question. RAG fetches the handful of passages most relevant to a question and drops them into the prompt, so the model answers from fresh, specific, private data it never saw in training.

Step 1: Chunk your documents#

Models read a fixed-size context, so you cannot dump a whole library into one prompt. Split each document into chunks of a few hundred tokens — very roughly a few paragraphs' worth of text. Overlap adjacent chunks slightly so a sentence stranded across a boundary still lands intact in at least one piece.

chunk_size = 400 tokens
overlap    = 50 tokens

Where you can, split on natural boundaries — paragraph breaks, headings, sentence ends — rather than slicing blindly every N tokens. A chunk that stops mid-sentence carries half a thought and embeds poorly. Fixed-size splitting is the crude baseline; structure-aware splitting that keeps each chunk semantically whole retrieves noticeably better.

Chunk too small and each piece loses the context that gives it meaning. Chunk too large and retrieval gets coarse — you pull in a whole page to answer a one-line question and waste prompt space. A few hundred tokens with light overlap is a sound default to tune from.

Step 2: Embed each chunk#

An embedding model turns a chunk of text into a vector: a fixed-length list of numbers that encodes its meaning. Passages about similar topics land near each other in this vector space. Run every chunk through a local embedding model — nomic-embed-text is a small, fast, widely used default that runs on a CPU; mxbai-embed-large (higher-dimensional vectors), bge-m3 (strong multilingual), and Google's embeddinggemma are common alternatives.

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "search_document: Your chunk of text goes here."
}'

The response contains an embeddings array — one vector per input. (The older /api/embeddings route, with a singular prompt and embedding, still works but has been superseded by /api/embed, which also accepts an array of inputs so you can embed a batch in one call.)

Two rules keep the vectors comparable. First, embed your query with the exact same model you used for the chunks — mixing models breaks the comparison, because two models do not place text in the same coordinate system. Second, respect your model's task prefix if it has one: nomic-embed-text expects search_document: in front of stored chunks and search_query: in front of queries, and dropping those prefixes quietly degrades retrieval even though the model is technically the same. Check your model's card; not every model uses prefixes, but several of the popular local ones do.

Step 3: Store the vectors#

Keep each chunk's vector alongside the original text and a reference back to its source. Each stored record needs three things:

• the embedding vector,
• the original chunk text,
• metadata such as the source document, title, and position.

The metadata is what lets you cite where an answer came from later, so do not skip it.

For a few thousand chunks, brute force is fine: hold the vectors in memory (or a file) and compare the query against every one. Past that, exact comparison gets slow, and you want a dedicated vector store that builds an approximate-nearest-neighbor (ANN) index — HNSW is the common one — to search millions of vectors in milliseconds. Local-friendly options include sqlite-vec, Chroma, LanceDB, Qdrant, FAISS, and pgvector if you are already on Postgres. The trade is exactness for speed: an ANN index can occasionally miss a true nearest neighbor, which for RAG almost never matters.

Step 4: Retrieve the nearest chunks#

At query time, embed the user's question (with the query prefix, if your model uses one), then find the stored vectors closest to it. Closeness is measured with cosine similarity, which compares the angle between two vectors rather than their length — so a short passage and a long one on the same topic still score as close. Return the top few matches, commonly three to five:

query_vector = embed("search_query: " + question)
top_k        = nearest_by_cosine(query_vector, stored_vectors, k=4)

Those top-k chunks are your evidence. top_k is your first tuning knob: retrieve too many and you flood the prompt with noise that dilutes the model's attention; too few and you risk missing the passage that actually holds the answer. (Many embedding vectors are normalized to unit length, in which case cosine similarity and a plain dot product rank results identically — a small but handy shortcut.)

Step 5: Assemble the prompt#

Now stitch the retrieved chunks into a clear instruction. The structure matters more than the wording:

Use only the context below to answer. If the answer is not
in the context, say you do not know.

Context:
<retrieved chunk 1>
<retrieved chunk 2>

Question: <the user question>

Send that to your local model — any small instruct model will do, and the same runtime that serves your generation model serves this prompt. Because the relevant passages sit right there in the prompt, the model can ground its answer in your data and quote specifics. The instruction to admit ignorance when the context lacks an answer is what keeps it from papering over gaps with invented facts.

One ordering detail earns its keep: models pay the most attention to the start and end of a long context and tend to skim what is buried in the middle — the "lost in the middle" effect. If you are packing in several chunks, put the strongest match first or last, not in the center.

Step 6: Show your sources#

Since every chunk carries metadata, append the source of each retrieved passage to the answer. This turns an opaque response into one a reader can check, and it is the single most valuable habit in any RAG system. It also makes retrieval auditable: when an answer looks wrong, you can see at a glance whether the model reasoned badly or the retriever simply handed it the wrong passage.

Beyond the basics#

The six steps above are a complete, working pipeline. Three upgrades matter once you outgrow it, in rough order of payoff.

Reranking. Embedding similarity is fast but blunt: it scores the query and each chunk separately, so "close in vector space" is not always "actually answers the question." A reranker — a cross-encoder such as bge-reranker — reads the query and a candidate chunk together and scores the pair directly. Retrieve a wide net with vectors (say the top 20), then rerank down to the 3 you actually send. It is the highest-leverage quality fix in most pipelines.

Hybrid search. Dense vectors are weak at exact strings — product codes, error numbers, rare names, acronyms — because those carry little semantic signal. Run a keyword search (BM25) alongside the vector search and merge the results, and you get both meaning and literal matches.

Query rewriting. Users ask terse or ambiguous questions. Having the model expand or rephrase the query before you embed it — or splitting a multi-part question into several retrievals — often surfaces passages a raw query would miss.

Where this breaks#

The failure most people hit first is bad chunking. If retrieval keeps returning passages that are almost-but-not-quite relevant, chunk size is usually the culprit — re-chunk with different sizes and overlap, and split on natural boundaries, before you touch anything else. If the chunks themselves are right but the ranking is still slightly off, that is the case reranking fixes.

The second is the embedding mismatch. Using one model to embed documents and another to embed the query puts the two in different coordinate systems and makes similarity meaningless. The subtler version is the same model with the wrong task prefix — technically matched, still degraded. Embed both sides with the same model and the same prefix convention.

Third, retrieval is only as good as what is in the store. If a fact was never ingested, no amount of prompt tuning will surface it, and the model will either say it does not know or quietly hallucinate. RAG extends a model's knowledge; it does not make the model omniscient. When an answer goes wrong, read what was actually retrieved before you blame the model — most "model" failures are retrieval failures wearing a disguise.