Vector Databases and Embeddings: What Executives Actually Need to Understand
There is a moment in modern AI procurement that repeats like clockwork: the vendor says "vector database", a slide appears with glowing dots, and the room nods. Nobody asks what the dots are, whether the organisation already owns the capability, or what the dedicated product costs at scale.
The nod is expensive. Vector infrastructure is genuinely the retrieval backbone of enterprise AI - and over-buying it is shaping up as this decade's version of the Hadoop clusters bought for spreadsheets.
Here is enough mechanism to replace the nod with three good questions.
Key Takeaways
The mechanism is geometry: meaning encoded as coordinates, similarity as distance, retrieval as nearest-neighbour search.
You may already own it: vector capability is embedded in platforms most organisations run today.
Dedicated engines pay at scale thresholds most mid-market workloads never reach - evaluate on criteria, not category.
How Do Vectors and Embeddings Actually Work?
An embedding model converts a chunk of text into a long list of numbers - coordinates in a high-dimensional space - positioned so that similar meanings land near each other. A vector database stores those coordinates and answers one question extremely fast: which stored points sit nearest to this new one? That nearest-neighbour search is retrieval by meaning.
This is why a search for "holiday rules" finds the "annual leave policy" despite sharing no words: both phrases map to neighbouring coordinates because they mean neighbouring things. No comprehension is involved - just geometry learned from how language is used.
Dimensionality, the spec sheets' favourite number, deserves demystifying: more dimensions mean finer distinctions of meaning at higher storage and search cost - and beyond a sensible range, the gains vanish into noise. Mid-market retrieval quality is won in content hygiene and chunking, almost never in dimension counts.
Two properties follow that executives should hold onto. First, the index is only as good as what was embedded - duplicates and stale documents get coordinates too, right next to the question. Second, distance is not permission: the geometry has no idea who may read the document behind a point, so security trimming must be engineered around the search, not assumed within it.
Do You Actually Need a Dedicated Vector Database?
Usually not at mid-market scale. Vector search is now an embedded capability in platforms most organisations already run - Azure AI Search, Microsoft Fabric, PostgreSQL with its vector extension, and the major data warehouses. Dedicated engines earn their place at high scale, high query rates or specialised filtering needs - thresholds a typical document estate never approaches.
The decision tree, honestly drawn:
Hundreds of thousands of chunks, moderate query volume: the embedded capability in your existing stack handles this comfortably - one fewer system to secure, license and operate.
Tens of millions of vectors, high concurrency, low-latency demands: dedicated engines justify themselves on performance and cost-per-query.
Heavy metadata filtering and hybrid search: evaluate specific products on those features - some embedded options are genuinely strong here, some dedicated ones surprisingly weak.
The procurement red flag is a proposal that begins with the engine rather than the workload. Infrastructure sized to the pitch deck instead of the use case is how shelfware happens.
What Are the Evaluation Criteria That Actually Matter?
Four criteria decide real-world retrieval quality, and none of them is raw speed: recall quality (does it find the genuinely relevant content?), permission-aware filtering (can results be trimmed to each user's access at query time?), hybrid search (combining meaning with keywords and metadata), and index freshness (how quickly content changes reach the search).
Note how the list mirrors data governance rather than database benchmarks. Recall is determined as much by content hygiene - deduplication, currency, sensible chunking - as by the engine. Permission filtering is an architecture decision spanning the source systems and the index. Freshness is a pipeline SLA. The engine is one component; the discipline around it is the product - which is why retrieval quality is treated as estate work in preparing your data for AI, not a purchasing decision.
Where Does Vector Search Fit in the Bigger AI Architecture?
It is the memory lookup inside larger patterns: RAG systems retrieve passages to ground answers, and agents retrieve context to inform actions. That placement means vector quality propagates - a weak index produces weak grounding produces wrong answers or wrong actions, regardless of how capable the model on top may be.
It also means the index inherits serious responsibilities the pitch rarely mentions: when retrieval feeds an agent that acts, permission-blind search becomes permission-blind action. Building the retrieval layer to the same standard as the agent's guardrails - scoped, trimmed, evaluated - is part of the engineering discipline of AI agent development rather than an infrastructure afterthought.
How Do Embeddings Get Created - and What Does That Cost?
An embedding model - distinct from the chat model - processes each content chunk once and emits its coordinates; you pay per token embedded, plus storage for the vectors and compute for searches. The economics are front-loaded and modest: embedding even a large document estate typically costs less than the meeting held to discuss it, and re-embedding happens only when content changes.
The cost structure worth understanding before procurement:
Initial embedding: a one-off pass over the estate, priced per million tokens - usually the smallest line in the project.
Incremental updates: only changed content re-embeds; a healthy pipeline embeds deltas nightly for pocket change.
Storage and search: vectors are compact, and mid-market query volumes run comfortably on embedded platform capability - the line that balloons only when a dedicated engine is bought prematurely.
The hidden line: chunking and cleaning labour before embedding. Garbage embedded is garbage retrievable; the preparation, not the geometry, is where budgets are honestly spent.
One choice does carry switching costs: the embedding model itself. Vectors from different models are not comparable, so changing models means re-embedding the estate. It is rarely expensive, but it is a migration - worth choosing a mainstream, well-supported model at the start rather than the benchmark leader of the week.
When Should You Re-Embed Your Content?
Three triggers, in practice. First, when you change embedding models - new model, new coordinate space, and old vectors are unreadable to it, so upgrades mean re-embedding the corpus. Second, when content changes - which is why pipelines re-embed updated documents automatically rather than on a calendar. Third, when you change chunking strategy, since how you split documents determines what each vector represents.
Budget accordingly: embedding cost scales with corpus size and is incurred again at each model migration. For most mid-market estates the dollars are modest, but the operational lesson is not - re-embedding is a standing pipeline capability you design for on day one, not a one-off project you repeat by surprise every time a vendor deprecates a model.
Three Questions Instead of a Nod
Next pitch, replace the nod with: What does our existing stack's vector capability lack for this workload, specifically? How is retrieval trimmed to each user's permissions, at query time? And what recall score does this system achieve on a test set built from our actual content?
Vendors with production systems will enjoy answering. The glowing-dots slide does not survive contact with the third question.
