Wooden library card catalog desk with an open drawer showing index cards, illuminated by a brass desk lamp

ArticlesoftwareDeep read

Vector Databases: How They Actually Work, and When You Don't Need One

Ahmad JAug 7, 20264 minUpdated Sep 15, 2026

A vector database does three plain things: store embeddings, index them, and answer "what's closest?" fast. Here's how that machinery really works (similarity metrics, HNSW and IVF indexes, quantization, filtering) and the many cases where Postgres already does the job.

A deep read: the full picture, with the receipts.

Signalstrong2independent sources

Vendors sell vector databases as mandatory infrastructure for anything with "AI" on the roadmap. They aren't. Strip away the pitch and a vector database does three unglamorous jobs: it stores embeddings, builds an index over them, and answers "what's closest to this?" quickly. That's the whole product. The intelligence you're actually paying for lives in the embedding model that produces the vectors, not in the store that holds them. Before you bolt a managed vector service onto your stack, it's worth understanding exactly what that store does, because your existing database may already do it.

What a vector embedding actually is#

An embedding is what you get when a model reads an object (a sentence, a paragraph, an image, a snippet of code) and emits a fixed-length list of numbers. That list is a coordinate: a single point in a space with hundreds or thousands of dimensions. The model is trained so that things which mean similar things land near each other, regardless of the exact words used. "The invoice is overdue" and "we haven't been paid yet" sit close together; "the invoice is overdue" and "the weather is nice" sit far apart.

Two points are worth nailing down, because they cause most of the confusion:

What people assume	What is actually true
The database makes the embedding	A model does: OpenAI's text-embedding-3, Cohere's embed-v3, Voyage's code-tuned models, Google's Gemini embeddings, or an open model you run yourself. The vector database just stores and searches what the model hands it. Swap the model and you have to re-embed everything
Vectors are vectors, so any two can be compared	Vectors from different models don't share a space. A 1,536-dimension OpenAI vector and a 1,024-dimension Cohere vector are not comparable, even if you pad them to the same length. Distances only mean something within one model's output

Dimensionality is a lever, not a fixed cost. Small models emit 384–768 numbers; larger ones go to 1,536 or 3,072 (text-embedding-3-large). More dimensions can capture more nuance but cost more to store and compare. Modern models increasingly support Matryoshka embeddings, where you can truncate a 3,072-dim vector down to 512 or 256 and keep most of the retrieval quality: a cheap, direct way to trade a little accuracy for a lot less storage and latency.

How similarity gets measured#

Once everything is a point in the same space, "similar" becomes "close." Three metrics dominate, and the differences between them are smaller than they look.

The metric	What it measures, and when the choice matters
Cosine similarity	The angle between two vectors, ignoring their length. It runs from `+1` (same direction, as similar as it gets) through `0` (perpendicular, unrelated) to `-1` (opposite direction). The correction that trips people up: `0`, not `-1`, is the "unrelated" case. Learned embeddings rarely point in opposing directions, so real scores cluster in a narrow positive band, which is why the ranking of scores matters far more than any single absolute value
Dot product	Multiplies the vectors component-wise and sums the result, folding in both angle and magnitude. The connection that simplifies everything: normalize every vector to unit length first and the dot product is the cosine similarity. Most pipelines normalize once at write time and then use dot product, because it is a single hardware-friendly pass with nothing to divide
Euclidean (L2) distance	Straight-line distance between the two points. For unit-normalized vectors it is monotonically related to cosine, so it ranks results identically. The choice only matters when magnitude carries real signal

The practical takeaway: normalize your vectors and cosine, dot product, and L2 largely converge. "Which metric?" is a much smaller decision than vendors imply.

Why exact search doesn't scale: and what ANN trades away#

The obvious way to find the nearest neighbors is to compare your query vector against every stored vector and keep the top matches. That's a brute-force scan: cost grows with the number of vectors times the number of dimensions. At a few thousand vectors it's instant and you should just do it. At tens of millions, under a latency budget, at real query volume, it falls over.

Approximate Nearest Neighbor (ANN) indexes fix this by giving up a sliver of correctness for a massive speedup. The correctness they give up is measured as recall: the fraction of the true nearest neighbors the index actually returns. Recall@10 of 0.98 means that, on average, the index finds 98 of every 100 true top-10 neighbors. Every ANN index is a recall-versus-speed dial, and both dominant designs let you turn it:

	HNSW (Hierarchical Navigable Small World)	IVF (Inverted File index)
The structure	A layered proximity graph: a skip list for geometry. Upper layers are sparse and let a search take long hops across the space, lower layers are dense for fine-grained final steps. A query starts at the top, greedily walks toward the target, and descends layer by layer	Clusters the vectors first, usually with k-means, into many cells each with a centroid. A query only searches the handful of cells whose centroids are nearest
The knobs	`M`, how many neighbors each node links to, and `ef_search`, how many candidates to keep in play during a query. Raise them for higher recall, lower them for speed	`nprobe`, how many cells to probe. Probe more for higher recall, fewer for speed
What it costs you	Fast and high-recall, but it holds the whole graph plus the vectors in RAM	Builds faster and uses less memory, which is why it is common at very large scale, often paired with quantization
Where it breaks down	It doesn't love churn: heavy inserts and deletes degrade the graph and eventually call for a rebuild	The approximation is structural: the true nearest neighbor can sit just across a cell boundary you never probed

That pairing points at the other big lever: quantization, or compressing the vectors so more of them fit in memory.

The method	What it does to each vector
Scalar quantization	Stores each number as an 8-bit integer instead of a 32-bit float: roughly 4× smaller
Binary quantization	Collapses each number to a single bit and compares with fast Hamming distance, up to 32× smaller, then re-ranks the survivors with full-precision vectors to recover accuracy
Product quantization	Splits vectors into chunks and replaces each chunk with a codebook entry
pgvector's `halfvec`	16-bit floats: the mild, barely-lossy version of the same idea

In 2026 this is how teams put hundreds of millions of vectors on a single machine without a specialized cluster, and it's a big reason "you need a dedicated database at scale" is less true than it used to be.

The part nobody demos: metadata filtering#

Pure nearest-neighbor search is the easy demo. Real queries almost always carry conditions: nearest neighbors where tenant_id = 42 and status = 'active' and created_at > last_week. Combining a filter with an ANN index is genuinely hard, and it's where a lot of real systems live or die.

There are two naive strategies and both have a failure mode.

The strategy	How it works	Where it falls over
Pre-filtering	Applies the `WHERE` clause first, then searches the survivors	Excellent when the filter is selective. When it isn't, you have thrown away the index and are back to scanning
Post-filtering	Runs the ANN search first, then drops rows that fail the filter	Fine until the filter is selective, at which point your top-k can come back nearly empty, because the neighbors the index found all got filtered out

This exact problem is why pgvector 0.8 introduced iterative index scans, which keep pulling candidates until enough rows pass the filter, and why every serious vector engine invests heavily in filtered search. If your workload is mostly filtered queries and multi-tenant isolation, this, not raw vector count, is the capability to evaluate.

When you don't need a dedicated vector database#

Most applications that reach for a vector database don't need one yet, and some never will.

The situation you are actually in	Why a dedicated store adds nothing
Prototypes and small corpora	Searching a few thousand to a few hundred thousand documents is comfortably a brute-force or single-index job. Provisioning managed infrastructure for it is pure overhead
Your data already lives somewhere capable	Vector search stopped being a product category and became a feature. Postgres has pgvector, and pgvectorscale for disk-backed scale; Elasticsearch and OpenSearch, MongoDB Atlas, Redis, ClickHouse and even SQLite, via sqlite-vec, all do nearest-neighbor search now. A separate vector store usually adds a sync problem, a second source of truth, and one more thing to operate, for a feature you already have
You are doing retrieval for RAG at ordinary scale	A Postgres instance with pgvector handling a low-millions corpus keeps your transactions, joins, filters and backups in one system. That is frequently the right architecture, not a compromise

pgvector in particular comfortably serves workloads into the millions of vectors. A common rule of thumb puts the point where teams start reaching for a specialized system somewhere around ten million, but that number bends hard with your dimensions, filtering, latency target, and how much RAM you can give the index. Treat it as a smell, not a threshold.

When one actually earns its keep#

Dedicated vector databases (Pinecone, Weaviate, Qdrant, Milvus, and the newer disk-first entrants) are real engineering, and there are workloads where they clearly win:

The workload	What the dedicated system gives you
Scale past what a general database wants to hold	Tens of millions to billions of vectors
Tight latency at high concurrency	Single-digit-millisecond retrieval under sustained query load, which needs purpose-built indexing, sharding and caching
Heavy filtered search, multi-tenancy and frequent updates	All of it solved for you at a scale where you would otherwise be tuning it by hand
Operational offload	Sharding, replication, index builds, quantization and failover handled by the service, so your team ships features instead of babysitting an index

The pattern to watch for: you are increasingly writing database features, persistence and filtering and replication and backups, around your vector index. That is the signal you have outgrown the bolt-on.

Databases vs. libraries#

It's worth separating two things the word "vector" gets slapped on:

What it is	Examples	What you get, and what you still have to build
Libraries	FAISS, hnswlib, ScaNN	Raw indexes that run inside your process and memory. Fast, free and dependency-light, but they hand you only the search: persistence, updates, metadata filtering, sharding, concurrency and crash recovery are yours to build. Perfect embedded inside a service or a notebook, as long as you know you are building the database around them. Annoy still works but is effectively legacy, and new work doesn't start there
Databases	Pinecone, Weaviate, Qdrant, Milvus	An index wrapped in everything a production datastore needs: durability, filtering, replication, auth, APIs, backups. You pay in money or in operations for not writing that yourself
The middle ground	Chroma and LanceDB (embedded but persistent), pgvector (an index living inside Postgres)	Where a lot of teams should honestly start, and often stay

A decision checklist#

Before adding a vector database, price it against these axes: the answer is usually "not yet":

The axis	The question to answer
Corpus size	Thousands, low millions, or tens of millions and up?
Latency and throughput SLA	Is anyone actually waiting on single-digit milliseconds at high QPS?
Filtering and multi-tenancy	How complex are the `WHERE` clauses riding along with each search?
Update rate	Mostly static, or high churn that punishes graph indexes?
Where your data already lives	Can you turn on vector search in a system you already run?
Operational appetite	Self-host and tune, or pay for managed?
Cost	At your real scale, not your imagined one

What this sets up#

Getting these mechanics straight is what makes a later comparison of specific services meaningful rather than a vibe check. Once you can reason about index type, recall targets, filtering strategy, quantization, and the operational gap between self-hosted and managed, vendor benchmarks stop being persuasion and start being data you can check. That's the ground the next piece builds on.

Takeaway: A vector database is a focused tool (store, index, and search embeddings) not a prerequisite for having an AI feature. The move from an embedded index or a Postgres extension to a managed service is a real step with real cost, and it's worth taking only when your scale, latency, and filtering demands actually force it. Until then, the boring option is usually the correct one.