Why Does Copilot Give Different Answers to the Same Question? (And How to Make It Stop)
- Matt Lazarus
- Aug 10
- 5 min read
The test takes thirty seconds and ends more AI rollouts than any security review. An executive asks Copilot for last quarter's revenue. Then asks again - new session, same words. Two answers come back. They are not the same number.
Adoption dies in that moment, and the post-mortem usually blames "AI randomness" - which is the one explanation that guarantees the problem never gets fixed, because most of the variance is not random at all.
Here is the taxonomy of why answers differ, and which causes you can engineer to zero.
Key Takeaways
Three variance sources, wildly unequal: sampling randomness is small; retrieval variance is large; definitional ambiguity is largest.
The biggest causes are yours to fix: conflicting sources and undefined metrics live in your estate, not the model.
Consistency is testable: same-question batteries should be a deployment gate, with a score leadership can watch improve.
Why Would an AI Give Different Answers to Identical Questions?
Three mechanisms, in ascending order of impact: generation sampling introduces small wording-level randomness; retrieval variance means different sessions can ground on different documents; and definitional ambiguity means multiple legitimate "correct" answers exist in your estate, and the system picks among them. The executive saw mechanism three and blamed mechanism one.
Understanding the ranking changes everything about the response. Sampling randomness - the dice-roll inherent in text generation - mostly varies phrasing, not facts, and can be constrained anyway. The numbers changing between sessions almost always traces to the deeper two mechanisms, both of which are properties of the data the system reasons over.
Which is the genuinely hopeful finding: the variance executives find most alarming is the variance most completely within the organisation's control.
How Does Retrieval Cause Inconsistent Answers?
Retrieval-grounded systems fetch the most relevant passages for each question and answer from them - and when the index holds duplicates, drafts and superseded versions, near-identical questions can retrieve different sources. Session one grounds on the current finance pack; session two, phrased a whisker differently, grounds on March's draft. Both answers are faithfully derived. They still disagree.
The retrieval fixes are estate hygiene, applied with intent:
Permission differences add a subtle wrinkle worth testing deliberately: two users asking the identical question can legitimately receive different answers when their access differs - correct behaviour that looks like inconsistency in a side-by-side demo. The battery should include same-user and cross-user runs so genuine variance and security trimming are never confused.
One authoritative copy: deduplicate the retrieval pool and retire superseded versions from the index - not just from the front page.
Authority signals: certified and current sources ranked above the long tail, so relevance stops being the only judge.
Freshness discipline: a defined reindex cadence, so the answer's currency is a known property rather than a coincidence.
Why Is Definitional Ambiguity the Largest Cause?
Because for many business questions your estate genuinely contains several defensible answers. "Last quarter's revenue" exists in the ERP one way, the sales system another, a finance workbook a third - differing on GST, intercompany and timing. No retrieval improvement fixes that: the system is choosing among truths, and it chooses plausibly, fluently and differently each time.
This is the variance source that prompting can never touch, because the contradiction is upstream of the model entirely. The fix is the single calculation authority: metrics defined once, certified, and computed by governed logic - with the AI routed to that authority for numeric questions rather than left to improvise from raw tables. The model narrates; the semantic layer calculates.
That architecture - one certified definition of every metric, serving dashboards, spreadsheets and AI identically - is precisely the work of semantic layer consulting, and it is the difference between an assistant that answers and an assistant that guesses well.
How Do You Prove Consistency Before Leadership Tests It?
Make consistency a deployment gate: build a battery of the questions executives will actually ask, run each many times across sessions and phrasings, and score the agreement. Deterministic questions - revenue, headcount, policy terms - should converge to identical answers; the consistency score becomes a tracked metric, watched release over release like uptime.
The battery does double duty as a diagnostic. When a question scores poorly, the failure attributes cleanly to its layer: answers citing different documents indicate retrieval work; answers citing the same sources but computing differently indicate definitional work; pure phrasing jitter indicates the (acceptable, constrainable) sampling layer. For Microsoft estates, that attribution exercise - alongside the permission and hygiene checks consistency depends on - is run formally within a Copilot Readiness Assessment.
Are Some Questions Allowed to Vary?
Yes - and drawing the line explicitly is part of the fix. Creative and generative asks (draft this email, suggest approaches, summarise for a new audience) are supposed to vary; deterministic asks about facts, figures and policy terms are not. Consistency engineering targets the second class, and user trust improves fastest when the system is honest about which class it is answering in.
The practical taxonomy for your test battery:
Deterministic: metrics, dates, policy clauses, entitlements - one correct answer exists; variance here is defect, full stop.
Bounded judgement: "which customers look at risk?" - the method should be stable and citable even where the framing varies; test that the same records surface for the same criteria.
Generative: drafting and ideation - variance is the feature; test for tone, grounding and policy compliance instead of sameness.
Communicating the taxonomy does real adoption work. When the rollout says plainly "numbers and policy answers are certified and consistent; drafting is creative and will vary", users calibrate their trust correctly from day one - instead of generalising one varied draft into "the AI is unreliable", or worse, one consistent number into "everything it says is certified".
The taxonomy also assigns the engineering budget honestly: certify the deterministic class first, because that is where the thirty-second executive test happens - and where it is winnable outright.
How Often Should You Re-Test Consistency?
On every meaningful change, plus a steady heartbeat. The triggers: content updates to the retrieval pool, index or embedding refreshes, model version changes, and any edit to the semantic layer's definitions. Each of these can shift answers without anyone touching "the AI" - which is precisely why consistency regressions surprise teams that only tested at launch.
The mechanism is a small regression suite: twenty to forty canonical business questions with certified expected answers, run automatically after each change and weekly regardless. Track the consistency score over time and alert on dips. It is the same discipline software teams apply to code - and it converts "the AI seems off lately" from a vibe into a ticket with a timestamp and a cause.
Assign the suite an owner the way you would any monitoring system. Consistency tests that nobody reviews decay into green-tick theatre within a quarter; a named steward who triages failures, retires stale questions and adds new ones as the business changes keeps the evidence honest - and keeps leadership's trust compounding instead of resetting.
Deterministic Questions Deserve Deterministic Answers
Business users hold AI to a reasonable bar: the same question about the same facts should produce the same answer, every time, for everyone entitled to ask it. That bar is fully achievable - not by taming the model, but by governing what it reasons over.
Fix the sources, certify the definitions, test the consistency. The executive's thirty-second test still happens - it just becomes the moment the rollout earns its trust instead of losing it.
