How LLMs Actually Use Your Data: Context Windows, Embeddings and Grounding Without the Mysticism
- Matt Lazarus
- Jun 22
- 5 min read
A curious thing happens in AI procurement meetings: million-dollar decisions are made using a vocabulary nobody in the room can define. The model "knows" things. It will be "trained on our data". It "understands" the business.
None of those phrases means what the room thinks it means - and vendors are rarely in a hurry to clarify.
Three mechanisms govern everything an LLM can do with your information: context windows, embeddings and grounding. Master these three and every AI proposal becomes legible.
Key Takeaways
Context windows are working memory: finite, paid by the token, and the model sees nothing outside them.
Embeddings turn meaning into coordinates, which is how systems find "annual leave policy" when you typed "holiday rules".
Grounding is the discipline that makes answers trustworthy - and "train it on our data" is rarely what you actually want.
What Is a Context Window, Really?
The context window is the model's working memory: the finite amount of text - your question, any documents attached, the conversation so far - that the model can consider while generating an answer. Anything outside the window does not exist for that response. The window is measured in tokens, and tokens are what you pay for.
The desk metaphor holds up: the model is a brilliant analyst at a desk of fixed size. You can place documents on the desk and it will reason across them impressively - but documents left in the filing cabinet contribute nothing, however relevant. A bigger desk (modern windows are large) helps, but every desk has an edge, and crowded desks degrade focus.
Two business implications follow immediately. Cost scales with how much you place on the desk per question - context is an operating expense, not a one-off. And someone, or something, must choose what goes on the desk - which is the retrieval problem, and the reason the next mechanism exists.
What Are Embeddings, and Why Should an Executive Care?
Embeddings convert text into coordinates in a high-dimensional space, positioned so that similar meanings sit near each other. They are how systems search by meaning rather than keywords - the mechanism that finds the "annual leave policy" when a user asks about "holiday rules", no shared words required.
The map metaphor: every chunk of your content gets a pin on an enormous map of meaning. A question gets a pin too, and the system retrieves the nearest content pins. No magic - geometry.
The map also explains a phenomenon that puzzles new users: the assistant finds an obscure document perfectly but misses an obvious one. Obscurity is not a concept geometry knows - only distance is. If the obvious document was chunked badly or duplicated five times, its pins sit in the wrong neighbourhood, and no amount of importance rescues a misplaced pin.
The executive-relevant consequences:
Retrieval quality is content quality. Duplicate, stale and contradictory documents all get pins; nearest-neighbour search will happily return the obsolete version sitting right next to the question.
Embeddings are not security. A pin on the map does not know who may read the document it points to - permission trimming must be engineered separately.
The map needs maintenance. Content changes; pins must be re-placed. The reindexing cadence is your answer-freshness SLA.
What Does Grounding Mean - and Why Is It the Trust Mechanism?
Grounding is the discipline of requiring the model to answer from evidence placed in its context - retrieved passages, certified definitions, query results - rather than from its general training. A grounded answer cites its sources; an ungrounded one is fluent recollection, with all the reliability that implies.
This is the mechanism that separates trustworthy deployments from impressive demos. The same model, asked the same question, behaves completely differently depending on what the surrounding system supplies as evidence: hand it the current policy and it summarises accurately; hand it nothing and it improvises from internet averages, confidently.
Grounding is also where data preparation earns its keep. Evidence can only be supplied if it exists, is current, is findable and is permitted - the precise checklist behind preparing your data for AI. And for numeric business questions, the strongest grounding is a certified calculation: the model narrates, a semantic layer computes.
Why Is "Train It on Our Data" Usually the Wrong Request?
Because training changes a model's skills and style, not its access to current facts - and it does so expensively, opaquely and without per-user permissions. What most organisations actually want is retrieval plus grounding: knowledge that updates nightly, respects access controls and produces citations. Training bakes a snapshot; grounding reads the live page.
There are legitimate fine-tuning cases - specialised vocabulary, consistent output formats, domain style - but they are the exception. The procurement red flag is a vendor proposing training as the answer to "make it know our business", because that answer is cheaper, safer and better served by the three mechanisms above.
How Do the Three Mechanisms Combine in a Real System?
Trace one question through a production assistant and the mechanisms snap together. A staff member asks "what is our parental leave top-up?" The system embeds the question, searches the index for the nearest content - the current policy, ideally - places the retrieved passages into the context window alongside the question, and instructs the model to answer only from what is on the desk, with citations.
Every quality property of the answer maps to one mechanism. Found the right policy? Embedding and index quality. Had room for the full clause including the exceptions? Context management. Refused to speculate when the policy was silent? Grounding discipline. The model's raw capability matters far less than the pipeline that feeds it - which is why two organisations with identical licences get unrecognisably different results.
The combination also explains the cost model executives should hold: every question pays for embedding the query (trivial), searching the index (cheap), and processing the retrieved context through the model (the real line item, scaling with how much lands on the desk). Tight retrieval that places three relevant passages beats generous retrieval that places thirty - in accuracy and in unit cost simultaneously.
One sentence to carry into the next vendor meeting: the model is the engine, but retrieval is the steering and your content is the road - and trips fail far more often from steering and road than from engine.
Does the Model Remember What You Told It?
Within a conversation, yes - that is the context window doing its job. Between conversations, no: the model's weights do not update because you used it, and tomorrow's session starts blank. What persists is not memory in the model but logging by the service - prompts and outputs retained under whatever terms your tier specifies.
The distinction dissolves two common confusions at once. The fear that "the AI is learning our secrets" is usually misplaced - enterprise tiers contractually exclude training on your data - while the comfort that "it forgets everything" is equally wrong if retention logging is switched on. The right questions are contractual, not mystical: what is logged, where, for how long, and who can see it.
Literacy Changes the Conversation
Once a leadership team holds these three concepts, AI proposals stop being acts of faith. "How much context does each query consume, and at what cost?" "What is in the retrieval index, and who curates it?" "Show me an answer's grounding." The vendors with real systems enjoy those questions. The others reschedule.
That asymmetry is the entire return on an afternoon of literacy.
