Vector Databases: How They Actually Work, and When You Don't Need One

Vendors sell vector databases as mandatory infrastructure for anything with "AI" on the roadmap. They aren't. Strip away the pitch and a vector database does three unglamorous jobs: it stores embeddings, builds an index over them, and answers "what's closest to this?" quickly. That's the whole product. The intelligence you're actually paying for lives in the embedding model that produces the vectors — not in the store that holds them. Before you bolt a managed vector service onto your stack, it's worth understanding exactly what that store does, because your existing database may already do it.

What a vector embedding actually is#

An embedding is what you get when a model reads an object — a sentence, a paragraph, an image, a snippet of code — and emits a fixed-length list of numbers. That list is a coordinate: a single point in a space with hundreds or thousands of dimensions. The model is trained so that things which mean similar things land near each other, regardless of the exact words used. "The invoice is overdue" and "we haven't been paid yet" sit close together; "the invoice is overdue" and "the weather is nice" sit far apart.

Two points are worth nailing down, because they cause most of the confusion:

• The database doesn't make the embedding. A model does — OpenAI's text-embedding-3, Cohere's embed-v3, Voyage's code-tuned models, Google's Gemini embeddings, or an open model you run yourself. The vector database just stores and searches what the model hands it. Swap the model and you have to re-embed everything.
• Vectors from different models don't share a space. A 1,536-dimension OpenAI vector and a 1,024-dimension Cohere vector are not comparable, even if you pad them to the same length. Distances only mean something within one model's output.

Dimensionality is a lever, not a fixed cost. Small models emit 384–768 numbers; larger ones go to 1,536 or 3,072 (text-embedding-3-large). More dimensions can capture more nuance but cost more to store and compare. Modern models increasingly support Matryoshka embeddings, where you can truncate a 3,072-dim vector down to 512 or 256 and keep most of the retrieval quality — a cheap, direct way to trade a little accuracy for a lot less storage and latency.

How similarity gets measured#

Once everything is a point in the same space, "similar" becomes "close." Three metrics dominate, and the differences between them are smaller than they look.

• Cosine similarity measures the angle between two vectors, ignoring their length. It runs from +1 (same direction — as similar as it gets) through 0 (perpendicular — unrelated) to -1 (opposite direction). Note the correction that trips people up: 0, not -1, is the "unrelated" case. In practice, learned embeddings rarely point in opposing directions, so real scores cluster in a narrow positive band — which is why the ranking of scores matters far more than any single absolute value.
• Dot product multiplies the vectors component-wise and sums the result, so it folds in both angle and magnitude. Here's the connection that simplifies everything: if you normalize every vector to unit length first, the dot product is the cosine similarity. Most pipelines normalize once at write time and then use dot product as the metric, because it's a single hardware-friendly pass with nothing to divide.
• Euclidean (L2) distance is straight-line distance between the two points. For unit-normalized vectors it's monotonically related to cosine, so it ranks results identically — the choice only matters when magnitude carries real signal.

The practical takeaway: normalize your vectors and cosine, dot product, and L2 largely converge. "Which metric?" is a much smaller decision than vendors imply.

Why exact search doesn't scale — and what ANN trades away#

The obvious way to find the nearest neighbors is to compare your query vector against every stored vector and keep the top matches. That's a brute-force scan: cost grows with the number of vectors times the number of dimensions. At a few thousand vectors it's instant and you should just do it. At tens of millions, under a latency budget, at real query volume, it falls over.

Approximate Nearest Neighbor (ANN) indexes fix this by giving up a sliver of correctness for a massive speedup. The correctness they give up is measured as recall — the fraction of the true nearest neighbors the index actually returns. Recall@10 of 0.98 means that, on average, the index finds 98 of every 100 true top-10 neighbors. Every ANN index is a recall-versus-speed dial, and both dominant designs let you turn it:

• HNSW (Hierarchical Navigable Small World) builds a layered proximity graph — think of a skip list for geometry. Upper layers are sparse and let a search take long hops across the space; lower layers are dense for fine-grained final steps. A query starts at the top, greedily walks toward the target, and descends layer by layer. Two knobs govern the trade-off: M (how many neighbors each node links to) and ef_search (how many candidates to keep in play during a query) — raise them for higher recall, lower them for speed. HNSW is fast and high-recall, but it holds the whole graph plus the vectors in RAM, and it doesn't love churn: heavy inserts and deletes degrade the graph and eventually call for a rebuild.
• IVF (Inverted File index) clusters the vectors first — usually with k-means — into many cells, each with a centroid. A query only searches the handful of cells whose centroids are nearest, controlled by nprobe. Probe more cells for higher recall, fewer for speed. The approximation is structural: the true nearest neighbor can sit just across a cell boundary you never probed. IVF builds faster and uses less memory than HNSW, which is why it's common at very large scale, often paired with quantization.

That pairing points at the other big lever: quantization, or compressing the vectors so more of them fit in memory. Scalar quantization stores each number as an 8-bit integer instead of a 32-bit float (roughly 4× smaller); binary quantization collapses each number to a single bit and compares with fast Hamming distance (up to 32× smaller), then re-ranks the survivors with full-precision vectors to recover accuracy; product quantization splits vectors into chunks and replaces each with a codebook entry. pgvector's halfvec (16-bit floats) is the mild, barely-lossy version of the same idea. In 2026 this is how teams put hundreds of millions of vectors on a single machine without a specialized cluster — and it's a big reason "you need a dedicated database at scale" is less true than it used to be.

The part nobody demos: metadata filtering#

Pure nearest-neighbor search is the easy demo. Real queries almost always carry conditions: nearest neighbors where tenant_id = 42 and status = 'active' and created_at > last_week. Combining a filter with an ANN index is genuinely hard, and it's where a lot of real systems live or die.

There are two naive strategies and both have a failure mode. Pre-filtering applies the WHERE clause first, then searches the survivors — excellent when the filter is selective, but if it isn't, you've thrown away the index and are back to scanning. Post-filtering runs the ANN search first, then drops rows that fail the filter — fine until the filter is selective, at which point your top-k can come back nearly empty because the neighbors the index found all got filtered out. This exact problem is why pgvector 0.8 introduced iterative index scans (keep pulling candidates until enough rows pass the filter) and why every serious vector engine invests heavily in filtered search. If your workload is mostly filtered queries and multi-tenant isolation, this — not raw vector count — is the capability to evaluate.

When you don't need a dedicated vector database#

Most applications that reach for a vector database don't need one yet, and some never will.

• Prototypes and small corpora. Searching a few thousand to a few hundred thousand documents is comfortably a brute-force or single-index job. Provisioning managed infrastructure for it is pure overhead.
• Your data already lives somewhere capable. Vector search stopped being a product category and became a feature. Postgres has pgvector (and pgvectorscale for disk-backed scale); Elasticsearch and OpenSearch, MongoDB Atlas, Redis, ClickHouse, and even SQLite (via sqlite-vec) all do nearest-neighbor search now. If your data is already in one of these, a separate vector store usually adds a sync problem, a second source of truth, and one more thing to operate — for a feature you already have.
• You're doing retrieval for RAG at ordinary scale. A Postgres instance with pgvector handling a low-millions corpus keeps your transactions, joins, filters, and backups in one system. That's frequently the right architecture, not a compromise.

pgvector in particular comfortably serves workloads into the millions of vectors. A common rule of thumb puts the point where teams start reaching for a specialized system somewhere around ten million — but that number bends hard with your dimensions, filtering, latency target, and how much RAM you can give the index. Treat it as a smell, not a threshold.

When one actually earns its keep#

Dedicated vector databases — Pinecone, Weaviate, Qdrant, Milvus, and the newer disk-first entrants — are real engineering, and there are workloads where they clearly win:

• Scale past what a general database wants to hold — tens of millions to billions of vectors.
• Tight latency at high concurrency — single-digit-millisecond retrieval under sustained query load, which needs purpose-built indexing, sharding, and caching.
• Heavy filtered search, multi-tenancy, and frequent updates at a scale where you want that solved for you rather than tuned by hand.
• Operational offload — sharding, replication, index builds, quantization, and failover handled by the service so your team ships features instead of babysitting an index.

The pattern to watch for: you're increasingly writing database features — persistence, filtering, replication, backups — around your vector index. That's the signal you've outgrown the bolt-on.

Databases vs. libraries#

It's worth separating two things the word "vector" gets slapped on:

• Libraries — FAISS, hnswlib, ScaNN — are raw indexes that run inside your process and memory. They're fast, free, and dependency-light, but they hand you only the search. Persistence, updates, metadata filtering, sharding, concurrency, and crash recovery are yours to build. Perfect embedded inside a service or a notebook; just know you're building the database around them. (Annoy still works but is effectively legacy — new work doesn't start there.)
• Databases — Pinecone, Weaviate, Qdrant, Milvus — wrap an index in everything a production datastore needs: durability, filtering, replication, auth, APIs, backups. You pay in money or operations for not writing that yourself.
• The middle ground — Chroma and LanceDB (embedded but persistent) and pgvector (an index living inside Postgres) — is where a lot of teams should honestly start, and often stay.

A decision checklist#

Before adding a vector database, price it against these axes — the answer is usually "not yet":

• Corpus size — thousands, low millions, or tens of millions and up?
• Latency and throughput SLA — is anyone actually waiting on single-digit milliseconds at high QPS?
• Filtering and multi-tenancy — how complex are the WHERE clauses riding along with each search?
• Update rate — mostly static, or high churn that punishes graph indexes?
• Where your data already lives — can you turn on vector search in a system you already run?
• Operational appetite — self-host and tune, or pay for managed?
• Cost at your real scale, not your imagined one.

What this sets up#

Getting these mechanics straight is what makes a later comparison of specific services meaningful rather than a vibe check. Once you can reason about index type, recall targets, filtering strategy, quantization, and the operational gap between self-hosted and managed, vendor benchmarks stop being persuasion and start being data you can check. That's the ground the next piece builds on.

Takeaway: A vector database is a focused tool — store, index, and search embeddings — not a prerequisite for having an AI feature. The move from an embedded index or a Postgres extension to a managed service is a real step with real cost, and it's worth taking only when your scale, latency, and filtering demands actually force it. Until then, the boring option is usually the correct one.