Podcast Summary: AI to ROI with Todd Olson (Pendo)

Episode Title: Measuring the performance and business impact of AI agents
Air Date: February 4, 2026
Host: Ray Rike
Guest: Todd Olson, Founder and CEO of Pendo

Overview

In this episode, Ray Rike speaks with Todd Olson about the real business impact of AI agents and how enterprises can effectively measure their performance. The discussion dives deep into AI agent analytics, measuring ROI, outcome-based pricing models, and the convergence of SaaS and AI. Todd brings forward both actionable advice for SaaS leaders and visionary perspectives on the future of agentic AI in digital products.

Key Discussion Points & Insights

1. The Evolving Need for AI Agent Analytics ([00:48]–[05:57])

• Background on Pendo:
  Todd explains how Pendo, originally focused on enhancing software experiences through embedded analytics, has evolved to apply its expertise towards AI agent analytics, enabling organizations to quantify ROI from AI investments.

  "Our goal is driving adoption of technology so that it ultimately honestly delivers on the ROI and promise, which I think connects very nicely with this podcast." — Todd Olson (01:18)

• Problems Enterprises Face:
  Organizations are mandated to experiment with AI but struggle to measure success beyond basic surveys, limiting meaningful improvements.

  "Honestly, the state of the art for a lot of the large, large enterprises we speak with is a basic survey... that’s not a way to make something better." — Todd Olson (05:35)

• Agent Analytics Vision:
  Pendo’s agent analytics solution collects conversational interactions, analyzes user engagement, identifies popular queries, retention rates, and frustration signals. The goal is to help teams understand and improve both user and agent performance.

  "By having that and processing that, we can start understanding what are the types of questions people are asking, how good are the responses, what percentage of your user base is leveraging [the agent]..." — Todd Olson (03:14)

2. Key Metrics for Agent Performance ([05:57]–[07:42])

• Out-of-the-Box Metrics:

  • Retention: Frequency of users returning to the agent for similar tasks.
  • Adoption: Percentage of users interacting with the agent.
  • Stickiness: How often users engage (consistency).
  • Frustration ("Rage Prompts"): Signals like repeated, agitated queries indicating poor agent experience.

  "If you can just eliminate people getting pissed off, I think you'll have a better experience." — Todd Olson (07:30)

3. Human in the Loop & Improving Experience ([07:43]–[09:20])

• Potential for Real-Time Escalation:
  While not live yet, Pendo is exploring triggering human assistance when frustration signals are high.

• Directing User Experience:
  Using prompt data to guide new users toward common or successfully-answered queries, with tailored suggestions.

  "It's not just like giving you metrics, it's how do we actually make the experience better?" — Todd Olson (08:45)

4. Outcome-Based Pricing in AI ([09:20]–[11:29])

• Challenges & Trends:

  • Clear outcomes (e.g., tickets resolved) work best.
  • For abstract outputs (like analytics), measuring value is harder.
  • Pendo provides data to support newer pricing models integrating usage and measurable outcomes.

  "We have really, really useful data for outcome-based pricing... It's going to be an interesting trend." — Todd Olson (09:53)

• Concrete Example: Predictions as a Service

  • Pendo Predict: Charges based on the number of predictions generated (e.g., churn or upsell predictions), aligning pricing with tangible business value delivered.

  "If you want to run the model more frequently, guess what, you're going to pay more because it's more predictions..." — Todd Olson (11:07)

5. Quality Predictions & Data Lineage ([11:30]–[14:56])

• Prediction Use Cases:
  Churn prediction, upsell likelihood, lead quality scoring—integrated directly into business processes (e.g., CRM notifications).

• Importance of Training on Customer Data:
  All predictions leverage unique customer datasets for relevance and accuracy.

  "By leveraging a machine learning model, it sort of strips the bias away and just lets... the numbers sort of drive the predictions." — Todd Olson (13:54)

• Tracking Data Lineage:
  Pendo supports tracking data flows across source systems to meet compliance and governance needs.

6. Agents Join the Org Chart ([14:56]–[17:30])

• Agents as Digital Team Members:
  AI agents increasingly automate roles (e.g., SDRs, localization tasks), freeing human teams for more creative and innovative work.

  "We have now agents that are helping our engineering teams handle certain tasks...that is dramatically shrinking and accelerating the way we build software." — Todd Olson (16:36)

• Acceleration of Product Development:
  Tools like V0 or Bolt are letting teams build prototypes and collect feedback in minutes/hours, not weeks.

7. Agent Performance Management ([17:42]–[19:22])

• Need for Performance Management:
  Agents, akin to human staff, require ongoing monitoring, feedback, and “coaching” to continually improve.

  "Agents need the same thing. They need coaching, they need feedback... The more context you have that you provide to an agent, the more accurate the results." — Todd Olson (18:30)

8. Advice for SaaS Leaders on Agentic AI ([19:22]–[21:19])

• Start with Painful, Repetitive User Workflows:
  Automate them, launch quickly, and use analytics to measure and improve agent performance.

  "What I've learned that speed is one of your most important currencies in this new AI economy. Like you just have to move really, really quickly." — Todd Olson (20:34)

• Measure Existing Product Usage:
  Strong analytics on current products lays the foundation for effective agent integration.

9. Adoption of Product Analytics ([21:19]–[22:38])

• Current State:
  Only ~30% of digital product companies use sophisticated analytics today—a missed opportunity for optimizing AI initiatives.

  "I bet you 30% are probably doing it ... If you’re thinking about this AI transition, a good place to start is just measuring your existing products and understanding what people are actually doing." — Todd Olson (21:52)

Notable Quotes & Memorable Moments

• On traditional measurement methods:
  "Honestly, the state of the art for a lot of the large, large enterprises we speak with is a basic survey. You know, we see people roll it out and they ask people, hey, was this successful or not? Yes or no? I mean that that's not a way to make something better." — Todd Olson (05:35)

• On the power of data-driven improvement:
  "It's not just like giving you metrics, it's how do we actually make the experience better?" — Todd Olson (08:45)

• On agents in the org chart:
  "We have now agents that are helping our engineering teams handle certain tasks... So that it ultimately unlocks more throughput and velocity for our teams, which is what gets me really, really, really excited." — Todd Olson (16:25)

• On advice for SaaS leaders:
  "Get something out there, automate a workflow, and then measure the heck out of it and understand what people want to do in your agent, because you’re going to learn a lot once you get it out there." — Todd Olson (20:02)

• On personal development for the AI economy:
  "We need to reflexively use AI tools because they're the future and honestly they're better than just simply Googling things... But then go check it, go check it." — Todd Olson (25:20)

Timestamps for Important Segments

| Segment | Topic | Timestamp | |---------|-------|-----------| | 1 | Introduction & Pendo’s background | 00:48–02:13 | | 2 | Agent analytics vision | 02:13–05:13 | | 3 | Pain points of measurement in enterprises | 05:13–05:57 | | 4 | Key metrics for agent performance | 06:19–07:42 | | 5 | Human-in-the-loop possibilities | 07:43–08:18 | | 6 | Improving agent experience | 08:45–09:20 | | 7 | Outcome-based pricing with Pendo Predict | 09:50–11:29 | | 8 | Example predictions & data lineage | 11:30–14:56 | | 9 | Agents in the org chart | 14:56–17:30 | | 10 | Agent performance management | 17:42–19:22 | | 11 | SaaS leader advice on agentic AI | 19:22–21:19 | | 12 | State of product analytics adoption | 21:19–22:38 | | 13 | Personal rapid-fire Q&A (AI use case, favorite tool, advice for grads) | 23:13–26:47 |

Final Takeaways

• Measuring and managing AI agent performance is critical for moving beyond AI “experiments” to real business value.
• Specific user analytics—retention, stickiness, frustration signals—are now being applied to agentic interfaces to drive ROI.
• SaaS leaders should move quickly, start with automating real pain points, and iterate with robust measurement.
• Personal adoption of AI tools and developing “reflexive use” is the key skill for the next generation workforce.

For more insights, check out Pendo's Agent Analytics and keep experimenting with AI tools to stay ahead in the rapidly evolving AI economy!