The misconception
You type a question:
“What were our top-performing channels last quarter?”
And you get:
- an answer
- a chart
- a clean explanation
No SQL. No data model. No friction.
The assumption:
The system understands your data.
What’s actually happening
Conversational analytics is not intelligence over your data.
It’s an interface layer within your data estate.
It translates:
- natural language → queries
- prompts → structured requests
- outputs → formatted results
In systems like BigQuery, this often includes agents that define how data should be interpreted.
But this is the critical point:
The system is not understanding your business.
It is interpreting your structure.
AI does not understand your data.
It relies on how your system defines it.
API / capability constraint framing
This interface is designed to answer specific types of questions.
It is not a general-purpose analysis layer.
It does not expand what your system can do.
It exposes what your system already supports.
What this interface is designed to return
Conversational analytics typically returns:
- a single query result
- a single breakdown
- a single visualization
It does not:
- reconcile multiple definitions
- combine conflicting logic
- perform multi-step analysis
It answers one question at a time—based on the structure it can access.
It doesn’t expand capability. It simplifies access.
The value of AI in analytics is not better answers.
It is faster access to answers—if the system is correct.
What people expect vs what actually happens
Expectation
- instant answers
- accurate interpretation
- consistent results
Reality
Everything depends on structure.
If your system is well-defined:
- answers align
- metrics are consistent
- outputs are reliable
If it isn’t:
- joins are incorrect
- definitions are misinterpreted
- results vary across queries
And the most important part:
The answers still look correct.
Direct queries vs structured agents
There are two ways conversational analytics operates.
1. Direct querying
- prompt → generated SQL → result
No context. No constraints.
This is fast—but unreliable.
The system guesses:
- which tables to use
- how to join them
- what your metrics mean
The same question can produce different queries—and different results.
2. Agent-based querying
Agents introduce structure.
They define:
- data sources
- instructions and defaults
- naming conventions
- verified queries
This improves accuracy.
But it’s important to be precise here:
Agents are not intelligence.
They are structured context.
They guide interpretation—but they do not fix broken systems.
If the underlying data is inconsistent, agents inherit that inconsistency.
Agents improve consistency—but they do not expand what the system understands.
What conversational analytics cannot do
It cannot:
- define your metrics
- resolve conflicting logic
- correct inconsistent data
- determine what matters
These must exist before the interface can function reliably.
What conversational analytics actually requires
This layer only works if the system underneath it is already structured.
1. Modeled data
Raw event data is not usable.
It must be transformed into structured tables.
If you skip this step, the system is forced to interpret raw exports.
For a deeper breakdown, see How GA4 BigQuery Export Changes Everything.
2. Defined logic
Metrics must be consistent across the system.
If logic varies by query, results will vary with it.
This is why logic must be enforced upstream—not recreated in prompts.
3. Semantic clarity
AI does not understand meaning.
It relies on it.
If your system contains:
- inconsistent naming
- ambiguous definitions
- unclear relationships
the system will guess.
And it will do so confidently.
This is where most systems fail.
4. Stable memory
Conversational analytics depends on querying stored data—not reconstructing it.
This requires a persistent system.
Without it, every answer becomes a one-off interpretation.
See BigQuery Vault.
Where conversational analytics fails
It doesn’t fail loudly.
It fails silently.
– Wrong joins
– Misinterpreted metrics
– Inconsistent outputs
– Partial data
All of these produce answers that look valid—but aren’t.
If this pattern is already happening:
AI doesn’t fix your data. It exposes it.
For a deeper breakdown, see Why AI Analytics Fails.
Where this fits in your system
Conversational analytics sits at the interface layer.
It does not:
- define your data
- enforce logic
- resolve inconsistencies
It depends on:
- data modeling
- semantic definition
- system structure
All of which exist upstream in your data estate.
If that system isn’t defined:
the interface cannot stabilize
What this enables (when it works)
Not better answers.
More reliable ones.
- consistency across queries
- alignment across teams
- outputs that reflect actual system behavior
This is the difference between:
- asking questions
- and trusting answers
Connection to AI-ready data
Conversational analytics is not where you start.
It’s where your system is tested.
If you’re evaluating these tools, you’re already at the interface layer.
The real question is:
Is the system underneath ready?
What to do next
If the answers are inconsistent, the issue isn’t the interface.
It’s the structure it depends on.
This only works if your data is structured
See AI-Ready Data
Have your system evaluated
Final principle
Conversational analytics doesn’t make your data understandable.
It makes your system visible.
And if that system isn’t structured:
the answers will still come back—just not reliably.
Doug McCaffrey
Designs and maintains analytics systems that remain reliable over time.
Explore how this connects across your data estate: