The misconception
AI analytics is being positioned as a solution.
Ask questions.
Get answers.
Move faster.
The assumption:
If AI can query the data, it must be accurate.
That’s not what actually happens.
What people expect
Most teams expect AI to:
- understand their business
- interpret their data correctly
- resolve inconsistencies automatically
The belief:
AI will make analytics easier—and more reliable.
What actually happens
AI does not improve your data.
AI moves complexity upstream into system design.
If your system is inconsistent:
- definitions conflict
- relationships are unclear
- logic varies across queries
AI will still produce answers.
And those answers will:
- sound correct
- look structured
- feel trustworthy
Even when they’re wrong.
This is where systems break.
Why AI analytics fails
AI fails for one reason:
it depends on a system that isn’t defined.
Everything that follows is a result of this dependency.
It is not interpreting your business.
It is interpreting your system.
1. Incomplete structure
If your data is not modeled:
- tables don’t align
- relationships are unclear
- queries produce inconsistent outputs
AI is forced to infer structure.
That inference is not stable.
For how structure is created, see Data modeling.
2. Undefined meaning
If your semantic layer is weak:
- metrics have multiple definitions
- naming is inconsistent
- relationships are ambiguous
AI has no stable interpretation.
It selects one—and proceeds as if it’s correct.
The same question can produce different answers depending on how it is interpreted.
For how meaning is enforced, see Semantic layer.
3. Fragmented logic
If logic lives in:
- dashboards
- queries
- individual tools
Then:
- definitions vary
- outputs conflict
- consistency disappears
AI cannot reconcile conflicting logic.
It reflects it.
For where logic should live, see Where logic belongs in a data estate.
4. Unstable systems
If your system drifts over time:
- tracking changes
- schemas evolve
- integrations degrade
AI operates on shifting inputs.
Outputs become unreliable.
AI systems require continuous refinement.
Without it, accuracy degrades over time.
For how systems degrade, see Technical drift.
The dangerous part
AI doesn’t fail obviously.
It fails silently.
Outputs can appear correct while being structurally wrong.
This is what makes AI failures difficult to detect.
1. Confident answers
Responses are:
- structured
- well-written
- definitive
They create trust.
2. Plausible outputs
Results look reasonable.
Even when they’re incorrect.
3. No clear errors
There are no visible failures.
No broken dashboards.
No missing data.
Just:
incorrect answers that look right.
Why this is difficult to detect
AI failures are:
- non-deterministic
- context-dependent
- presented with confidence
Which makes them harder to identify than traditional reporting issues.
AI does not guarantee correctness.
It guarantees an answer.
Because of this, AI outputs must be validated. Not assumed.
Direct queries vs structured systems
AI can operate in two ways.
1. Direct querying
- no context
- no constraints
- no definitions
Fast—but unreliable.
2. Structured systems
- modeled data
- defined meaning
- enforced logic
Slower to build—but stable.
AI only works reliably in the second case.
In direct querying, results are not guaranteed to be consistent—even for the same question.
In structured systems, outputs become predictable.
Where this fits in your system
AI sits at the interface layer.
It does not:
- define structure
- enforce logic
- create meaning
It depends on:
- data modeling
- semantic definition
- system design
If those layers are unstable:
outputs cannot be reliable
What prevents failure
AI analytics doesn’t need better prompts.
It needs a defined system.
Structured data
Predictable tables and relationships
Defined meaning
Consistent metric definitions
Enforced logic
Centralized, reusable calculations
Stable memory
Reliable, queryable data over time
This is what makes data usable.
Not AI.
Connection to AI-ready data
AI-ready data is not about adding AI.
It’s about preparing the system AI depends on.
If the system is structured:
- outputs align
- results stabilize
- trust becomes possible
If it isn’t:
AI amplifies the problem
What to do next
If AI outputs are inconsistent, the issue isn’t the tool.
It’s the system.
Fix the system, not the interface
See AI-ready data
Evaluate your system
See Evaluate
Final principle
AI doesn’t break your data.
It reveals how your system actually behaves.
And if your system isn’t defined:
the answers will still come back—just not reliably.
Doug McCaffrey
Designs and maintains analytics systems that remain reliable over time.
Explore how this connects across your data estate: