How Agentic AI is changing your analytics and what do to about it
A strategic framework for performance, optimisation, and trust
As organisations rapidly integrate GenAI and autonomous agents into their digital products, traditional product analytics - centered around linear funnels, click-through rates, and session duration - are proving insufficient to capture value or manage risk. We are witnessing the rise of a silicon-based workforce1 where AI agents proactively plan, execute, and reason to achieve user goals.
While the economic potential of Agentic AI is projected to reach $52.62 Billion by 20302, the primary barrier to realising this value is trust. With AI projects facing failure rates up to 80%3, consumer trust has become a primary barrier to value, meaning product analytics must expand from performance monitoring to becoming the trust architecture of the enterprise, bridging the gap between technical capability and human reliability in an AI-driven world.
From linear funnels to agentic loops
Traditional digital analytics was built for a world of finite, pre-determined choices. In such a world, the user's path is bound by navigation menus and information architecture. Here, analytics becomes the study of attrition along a pre-defined linear path.
But in a world of conversational AI and autonomous agents, the model breaks. Because users no longer navigate structures; they declare intent. And agents no longer follow hard-coded scripts; they reason and act dynamically.
Consider how a user might prompt: "Book me a flight to London under $600 and find a hotel near the conference centre." In one conversational turn, the agent traverses what used to be four distinct funnels across two verticals, calling multiple external tools.
“With Agentic AI, the linear analytics model breaks, and it becomes clear that we must move from mapping actions to mapping intent and reasoning.”
So how do we build an effective setup for agentic analytics?
It requires that we understand the reasoning loop (popularised by the ReAct framework): the process where an AI agent repeatedly observes, plans, acts, and evaluates across multiple steps.
Where traditional analytics tracked clicks and page views, agentic analytics tracks prompts, turns, and tool calls. Journeys shift from linear trees to cyclical, non-deterministic graphs. Success metrics move from conversion rates to containment rates, resolution quality, and sentiment. And the primary risk is no longer drop-off, but hallucination, misalignment, and loop-lock.
This means that human judgment and ability to navigate semantic complexity will continue to play a central role. According to Deloitte’s 2025 Connected Consumer Survey, consumers are more likely to engage with - and pay for - tech experiences they trust4. It is not enough for an agent to be fast; it must be trusted.
In that sense, analytics must capture Trust Signals: Did the agent proactively offer relevant suggestions? Did it respect the user's data privacy preferences? These are becoming hard KPIs influencing retention and brand perception.
So how do you build an analytics backbone that can support this new paradigm of agentic loops?
The architecture of agentic observability
To support the metrics and signals of the agentic loop paradigm, you need an architecture suited for much larger volumes of data than previously. The density and dimensionality of LLM logs (inputs, outputs, reasoning steps, tool outputs, embeddings) are significantly more complex than traditional clickstream data.
The MELT framework for LLMs
While technologies like ClickHouse, ElasticSearch, and BigQuery become indispensable for compressing and querying high-cardinality log data at speed, organising data becomes an equally vital task.
To organise this data, we adapt standard IT observability's MELT framework (Metrics, Events, Logs, Traces) for GenAI:
- Metrics: Tokens per second, cost per interaction, cache hit rates, latency, and throughput.
- Events: Tool invocations, guardrail triggers, and user feedback (explicit thumbs up/down or implicit signals like copy-pasting answers).
- Logs: Exact prompts and completions, plus audit trails recording access and decisions, crucial for EU AI Act compliance.
- Traces: End-to-end request journeys across distributed systems (for example, UI → orchestrator → LLM → vector database → external API → back), critical for debugging multi-agent latency.
Visualising the invisible
Standard dashboards cannot capture the non-linear nature of agentic conversations. We recommend:
- Circular Sankey Diagrams and Network Graphs that dynamically map conversation paths based on semantic clusters, grouping "I want a refund" and "Give me my money back" into a single flow node to reveal hidden intent flows.
- State Transition Diagrams modelling the agent as a state machine to highlight trap states where the agent gets stuck in loops.
The Conversation Funnel
The Conversation Funnel replaces the static page funnel as the unit of measurement. It tracks dialogue health from Intent Recognition to Resolution:
“In a linear funnel, users drop off between steps. In a conversation funnel, users loop between steps.”
These loop-backs often indicate confusion or poor slot-filling design. Key metrics to watch include Intent Recognition Rate, Sentiment Decay (how sentiment degrades during loops), Containment Rate (targeting 70-90% for advanced bots), and Time to Intent Resolution, the speed of value metric.
Performance, benchmarking, and optimisation
As autonomous AI agents transition from tools to decision-making members of organisations, performance measurement must focus on evaluating their competence as digital workers. We categorise metrics into four levels:
Benchmarking can be challenging because agents are non-deterministic. Organisations are adopting LLM-as-a-Judge frameworks, where a stronger model evaluates production outputs against a golden dataset of ideal responses. A key trade-off is speed versus intelligence: fast and simple agents suit L1 support, while slow and smart agents handle complex advisory roles.
The optimisation loop closes when user feedback (a “thumbs down" or an edited draft) triggers immediate log entries tied to conversation traces. Automated systems analyse failure clusters to refine system instructions, creating continuous improvement.
Trust, safety, and governance
“Trust is the currency of the Agentic Era.”
Hallucination and groundedness
Analytics must employ automated grounding checks using Natural Language Inference (NLI) models to verify responses against retrieved context, and measure citation accuracy.
Auditing autonomous agents
As agents move from assistive to autonomous, particularly in high-risk environments, liability shifts to the enterprise:
- The "Flight Recorder" Concept: Every agent decision must be logged with a reasoning trace.
- Compliance by Design: The EU AI Act requires high-risk AI systems to be traceable. Auditors must be able to replay conversations and see exactly what data was accessed.
- Permissioning Analytics: Anomaly detection monitors whether agents attempt to access systems beyond their scope.
Human experience in the age of Agentic AI
Sentiment is not static; it is a trajectory. The frustration curve tracks how rapidly sentiment drops, triggering escalation to a human. The tone delta measures alignment between user and agent communication styles.
To ensure agents are trustworthy, we rely on Deloitte's Trustworthy AI™ framework with three key metrics:
- Anticipation Ratio: How often the agent proactively offers useful information before being asked.
- Forgiveness Factor: How quickly sentiment recovers after an agent error.
- Transparency Score: Whether users understand why the agent made a decision.
The path forward for product analytics
The transition to Agentic AI is the most significant shift in digital product analytics since the invention of the cookie. We are moving from observing clicks to understanding intent and reasoning.
The organisations that master conversation analytics will not just optimise conversion rates; they will redefine the relationship between human intent and digital execution.
Click here to explore our AI Customer Agents Series: a series on crafting modular AI Customer Agents that deliver tangible value.
Sources:
1. Deloitte’s Tech Trends 2026. Deloitte. December 2025.
3. Harvard Business Review, Keep your AI projects on track: Most go off course. To make sure yours succeed, consider these five steps. November 2023.
4. Deloitte’s 2025 Connected Consumer Survey. Deloitte. September 2025.
