What is a Vector Database?

Vector Databases: Where They Actually Earn Their Keep

You can bolt a vector approach onto your relational database as you implement AI use cases in your organizational environments.

With a new database type showing up every few years, it has become common for half of any conference session on that new type to be spent redefining the term for a new audience. Right now vector databases are having their time in the sun, and everybody seems to need one for their RAG pilot or their AI-assisted service in the enterprise. Below the level of the slides and the conference talks, though, vector databases are not terribly complicated. An embedding model turns input data — text, images, support tickets — into a vector of numbers. Those vectors get stored. At query time, you embed the query the same way and search the stored vectors for the ones closest in meaning.

What it actually is

So first off, let me define it. A vector database is a database where you store vectors in a high-dimensional space — think 50, 100, 1,000 dimensions or more — and where you can query for the vectors most similar to a given vector in that space. There are broadly two strategies for getting data into that space: single-shot approaches, where an entire item is mapped to one vector, and chunking approaches, where the data is split into pieces and each piece is embedded separately.

In terms of suitability, vector databases fit retrieval problems where many items in a dataset are all somewhat like one given item. They are a poor fit for problems where you need to find the single row matching an exact set of criteria. That second category is what relational databases have always been good at. Postgres is very well suited to finding one specific piece of data given criteria that match on every field — think where user_id = 4471. That is in sharp contrast to what a vector database does, which is to hand you back the twenty items most similar to a given item.

Both kinds of system sit on top of storage engineered for a high volume of lookups, so in that sense they are cousins. The difference is in how the lookup happens. A relational database evaluates a set of criteria against the fields of each candidate row to determine a match. A vector database compares a query vector against the stored vectors — all of them, or more often a subset — using a distance metric, and returns the closest ones. Which metric you use depends on the nature of the data, but cosine similarity and Euclidean distance are the usual suspects.

Most vector databases use approximate nearest neighbor (ANN) indexes, such as HNSW or IVF, to make the search efficient. The primary trade-off with these indexes is precision against performance: the index may return vectors that are slightly less similar than the true nearest neighbors in exchange for returning them much faster. At very large scale — hundreds of millions or billions of vectors — even a small amount of additional retrieval time makes a real difference, so giving up a little accuracy is often an acceptable price for a real-time interactive application.

Where the engineering complexity actually lives

I think the biggest misperception about the engineering required for vector databases is that the hard part is the storage. It isn't. It's the indexing. There are a number of approaches, each with its own trade-offs, and a great deal of engineering goes into wringing performance out of them.

HNSW (Hierarchical Navigable Small World). Builds a multi-layer graph in which each vector is connected to its neighbors. A search starts in the coarse upper layers and descends into progressively finer ones. It is very fast and, for many applications, very accurate — which is why it is the default index in a lot of vector databases handling real-time interactive workloads.

IVF (Inverted File Index). Organizes all the vectors into partitions — buckets or clusters — centered on a set of centroids. At query time, you identify the clusters of interest and only traverse the vectors inside them. You give up some precision for a given recall target in exchange for speed. At very large scale (100M to 1B+ vectors), narrowing to a subset before running the similarity search is frequently the only remaining lever you have. IVF variants differ in how they structure partitions and how they select centroids, which is what lets the approach cover a lot of different use cases.

Product quantization. Compresses the stored vectors to reduce memory footprint and to cut the number of operations needed for each distance computation. It is very often paired with IVF — IVF-PQ — as the retrieval strategy for huge datasets.

Flat, brute-force scan. The simplest method of all: no index, just compare against everything. Worth remembering that brute force is exact — it never returns a wrong answer; it just gets slow. That makes it the right call more often than people assume, for small collections or anywhere exact results genuinely matter more than latency.

None of these is better than the others in any absolute sense. Each represents an engineering trade-off between latency, recall, and memory for a specific use case. Saying HNSW is the best index for every problem is about as sound as saying every workload belongs on the fastest disk tier.

Where retrieval quality actually comes from

Here's the core of the problem, and I'll defend it: the index is not what's failing your RAG system. The chunking and embedding strategy is.

The quality of the index depends entirely on the quality of what went into it — chunks of a document, embedded into a vector space by an embedding model. At retrieval time, the same model has to embed the query so the two can be compared. If the model that indexed the corpus and the model that embeds the queries are not the same model, you are comparing coordinates from two different spaces that happen to have the same number of dimensions. The similarity scores that come back will look perfectly reasonable and mean absolutely nothing. Swapping embedding models mid-project causes a great deal of pain for exactly this reason.

Two things I've seen bite teams repeatedly:

• Naive token splitting underperforms semantic splitting. Cutting every N tokens regardless of structure produces chunks that straddle two unrelated ideas. Split on paragraphs, headings, or logical sections instead.
• Embed consistently. Use the same model for your corpus and your queries. Embedding models are defined in terms of a coordinate space, and two models with the same dimensionality are not defined in the same coordinate space. Swapping one for the other is switching to an entirely different space, and retrieval breaks completely — silently.

It is also very useful to store metadata alongside the vectors: source, date, tags, owner, department. That metadata is what lets you filter candidates out before the similarity search runs, or filter results after it returns. A lot of teams realize too late that retrieval is the part that matters, and that a nice demo can hide an enormous number of problems.

Security, or why retrieval carries a false sense of safety

This is the part of the architecture everybody wants to skip, because storing derived numbers feels safer than storing sensitive text. It isn't. As I alluded to above, it is often possible to reconstruct or infer meaningful information about the original input from the embedding itself, particularly when the model is known.

So treat the vector store with the same rigor you would apply to any production database holding regulated data. That means access controls, tenant isolation, encryption at rest and in transit, and deliberate logging discipline around both query vectors and retrieved results — the queries themselves are a sensitive signal, not just the documents. Enforce metadata filtering before retrieval rather than bolting it on afterward. And in a regulated environment, keeping the whole retrieval layer inside your own VPC is usually the difference between a straightforward compliance conversation and a painful one.

Where this shows up in practice

The dominant use case at the moment is supporting a generation model: embed a large corpus, store the vectors, retrieve the relevant chunks for each query, and pass them to the model as additional context. But it doesn't stop at semantic search — the same mechanics power recommendation, anomaly detection, and deduplication. Here is how this tends to land in three domains I get asked about often enough to be worth spelling out.

Higher education. For a university's AI advising assistant to be useful to students and staff, it has to retrieve across financial aid policy, degree requirements, and course catalogs from every department. Financial aid policy changes year over year and by cohort, and it varies with in-state or out-of-state status, scholarships, and professional programs. So chunk that content by policy section and effective date rather than by PDF page break, or you will confidently serve a 2019 refund policy in answer to a 2026 question. The access control point from the previous section is not theoretical here either: a student must not have a path, even an indirect one, to another student’s advising notes or FERPA-protected records simply because their embeddings landed near each other in the vector space. Filtering by student ID and role before the similarity search runs is what keeps this a retrieval system rather than an incident report.

Finance. Research desks and trading floors accumulate documents continuously — vendor research, internal risk memos, filings — over long periods, with vendors added and dropped along the way. Two failure modes show up here reliably. The first is the model swap: a collection is built with one embedding model, someone later substitutes a comparable one, and answer quality quietly collapses while the similarity scores continue to look flawless. The second is staleness. Without collection-level metadata — when a document entered the corpus, how many revisions it has been through, whether it has been superseded — an analyst will eventually get handed a three-year-old note with no indication that it no longer reflects the house view. Filtering on that metadata before retrieval is the fix. And logging discipline matters twice over in this domain, because query vectors over trading strategy documents and client portfolio data are themselves sensitive.

Healthcare. Consider semantic search over clinical notes and discharge summaries for a clinician-facing assistant. Clinical narrative is exactly the kind of narrow, structured, sensitive text where reconstruction risk against embeddings is highest, so PHI-bearing vectors belong behind the same authentication and audit controls as the EHR itself — not a lighter-weight index sitting alongside it. On the retrieval side, chunking by clinical encounter and tagging each note with department and note type (history and physical, progress note, consult) is what keeps a nurse's query about a medication history from surfacing that patient's psychiatric notes. That is as much a chunking and metadata design decision as it is a policy one.

The Takeaway

A vector database is more than a search index bolted on top of your stack. It is part of your data layer, and it deserves the same rigor you would give any production system holding data your organization is accountable for. The indexing algorithm is the interesting part, but the chunking strategy is the part that decides whether retrieval is any good, and the access controls on the retrieval layer matter every bit as much as the ones on the rest of your data layer. Get the interesting part right and skip the other two, and you will have built something fast, confident, and wrong.

Dr. Sam Kurien