Podcast Summary: The Analytics Power Hour

Episode 284: I Used to Think… But Not Any More
Release Date: November 11, 2025
Hosts: Michael Helbling, Moe Kiss, Tim Wilson, Val Kroll, and Julie Hoyer

Episode Overview

This episode is a lively, candid roundtable where the hosts share and reflect on analytics beliefs they've outgrown over their careers. Inspired by an interview question (“What is a deeply held belief you’ve changed your mind on?”), the group offers a confessional look at shifting perspectives on core topics like attribution, data collection, organizational maturity, privacy, reporting, and engagement with stakeholders. They poke fun at themselves, revisit old mantras, and demonstrate the value of continuous learning—a must-listen for anyone in data and analytics seeking wisdom beyond best practices.

Key Discussion Points & Insights

1. The Origins: Changing Beliefs as a Marker of Growth

• Val Kroll introduces the episode’s theme, recalling a meaningful interview question about changing one's mind—a signal of intellectual humility and adaptability.

  “If you have someone that says like, 'Well, I've never changed my mind about anything,' that can be a good checking the box on 'does not fit culture here at whatever company.'” (02:22, Val)

• Michael Helbling ties this reflection to industry growth, noting the analytical profession is ever evolving with new priors and knowledge.

2. Attribution and Data Collection Fallacies

The Single View of the Customer

• Moe Kiss (05:03):

  “I still wrestle with... this idea of the single view of the customer… If you collect all these events, we will have a complete and perfect picture and be able to understand what our users want. And I especially think that was true in the world of attribution.”

• The group discusses the slow erosion of confidence in attribution models, shifting from faith in algorithmic “truth” to recognizing overconfidence and fundamental limitations.

The Attribution “Weights” Revelation

• Michael Helbling recalls a breakthrough moment:

  “They got to this one point where they’re like, 'and here’s where we just put in the weights we want for each channel.' And I was like, so that's not attribution. That's just adding things up to 100 based on whatever you feel like.” (06:41)

The Potato Chips Analogy

• Tim Wilson and Moe revisit analogies to illustrate the absurdity of user-level attribution:

  “Somebody is buying potato chips... They're already in their cart. Someone jumps in front... says 'you should buy these potato chips.' ... You're going to get some credit because you were the last touch…” (10:00, Tim)

Memorable Moment: This lightbulb moment (and the extended chip/pizza shop analogy, 09:49–10:50) underscores how much initial faith the panel once put in faulty methodologies.

3. Organizational Models & Analytics Maturity

Moving Beyond Centers of Excellence

• Val shares a major shift:

  “I used to think that there was this optimal model of operating inside an organization and that a center of excellence meant that you were the most mature...” (11:39)

• The team now recognizes organizational design (centralized vs. decentralized, CoE vs. hub-and-spoke, etc.) is not a fixed indicator of maturity; it’s always context dependent.

Maturity Models: Linear vs. Real-World Progress

• Tim and Michael critique classic maturity models for their oversimplifications:

  “These maturity models always assume sort of like this linear function and no organization works that way... It never rang true whenever I would sit down and try to apply a model... Eventually it started being like, 'Oh, pull back a little bit and realize this doesn’t just... work in the real world this way.'” (16:04, Michael)

• Val: “If I… wrote the questions for a maturity survey would look very different than they are today. Right? Like today it’s would be things about like, how are you ensuring analytics moves at the speed of the business…” (18:49)
  Everyone agrees: There is no “analytics nirvana.”

4. Privacy Attitudes: “No One Actually Cares” to Privacy By Design

• Michael confesses his early view:

  “In episode 10, way back in 2015, I said, 'Nobody actually cares about privacy.' And at that point in time, that was kind of true. But I think we’ve all been made to care.” (20:42)

• Tim Wilson reflects on the American vs. European privacy experience:

  “I have radically changed my belief. I think we fetishized the user level tracking to the point that we get caught up in… multi touch attribution… There’s lots of value we can get without violating privacy...” (23:30)

5. Data’s Role in Decision-Making (“Part of the Meal”)

• Moe Kiss:

  “I feel like I definitely fell prey to this… that data was right or that if you made good decisions it was because you were using a data-informed decision. Now… data is part of the meal.” (27:57)

• The meal analogy (data as one dish, not the whole spread) resonates, pushing back against the fallacy that all good decisions are exclusively data-driven.
• Tim: “It never has been and it never will be... It’s not even a progression… People are on [a train] that is totally misguided.” (30:23)

6. The Limits of Raw Insight Mining and Reporting

• Michael:

  “I used to think that you could just sit down with your data and come up with insights… just spend time with your data and you’ll just come up with cool stuff. And… I learned like, no, you can’t just sit in front of a dataset and [expect] valuable insights.” (33:11)

• Tim and Moe recount awkward past practices where analysts were expected to “find something” in regular meetings—fueling pointless “insight theater.”

  “He was like, look, we have a biweekly meeting. We got to shuffle through the data. We got to find something new to show them every time.” (34:37, Tim)

7. Ticketing Systems vs. Relationship-Based Inquiry

• Val admits past faith in ticketing systems as a solution to manage analytics requests:

  “I believed that having, like, a ticketing system… was a great front end to the analytics experimentation team… But I’ve definitely evolved on this.” (37:19)

• Tim retorts:

  “You’ve put a structure in front of them that implies that if they put this in, you will be good. And… you’ve already put a burden on them… You’re actually making it harder on the analyst [to probe deeper].” (43:34–45:17)

• Mo and Val provide counterpoints: forms introduce productive friction and help prioritize, especially for larger data teams or junior analysts.

• Tim: “I think almost everything else would be much, much better served by having a thoughtful coming together of what are we really trying to do here.” (48:52)

8. Should We Test (or Track) Everything?

• Hot Take Round (53:06):
  • Val: “Thumbs down.”
  • Michael: “People who think you should track everything also think you should test everything.”
• The panel unanimously agrees: indiscriminate testing or tracking has been replaced by a more nuanced, business-aware approach.

Notable Quotes & Moments

• On the illusion of single source of truth:

  “I used to believe that shit mattered and was going to give me better measurement.” (04:28, Michael)

• On privacy and data value:

  “I have radically changed my belief. I think we fetishize the user level tracking… There’s lots of value we can get without violating privacy or even getting close to it.” (23:30, Tim)

• On data’s true place:

  “Data is part of the meal… sometimes it’s the roast potatoes, sometimes it’s just the garnish.” (27:57, Moe)

• On ticketing systems:

  “You’ve put a structure in front of them that implies that if they put this in, you will be good… And then you’re coming back and wanting to engage with me. You’re making it harder.” (43:34, Tim)

• On insight mining:

  “I used to think you could sit down with your data and come up with insights... but really what we were doing was a little more nuanced.” (33:11, Michael)

Timestamps for Important Segments

• 02:22 – Interview question origin of episode theme
• 05:03 – The quest for the single view of the customer
• 06:41 – Attribution as self-justification; realization about “just putting in weights”
• 09:49–10:50 – The potato chips and pizza shop analogies riff
• 11:39 – Val’s shift from “Center of Excellence” as the only mature model
• 16:04 – Critique of linear maturity models (Michael)
• 20:42 – Michael’s about-face on privacy and the early episode 10 quote
• 23:30 – Tim on the fetishization of user-level tracking and privacy
• 27:57 – Moe’s meal analogy for decision making with data
• 33:11 – Michael’s evolution on “insight hunting”
• 43:34–45:17 – Ticketing systems debate between Tim and Val
• 53:06 – Lightning round: "You should test everything" is a myth

Overall Tone & Takeaways

Authentic, irreverent, and intellectually honest, this episode invites listeners to recognize the fallibility and evolution natural to analytics careers. The hosts laugh with and at themselves, share war stories and failed convictions, but come back to deeper truths about data’s role, humility, and business partnership. Longtime practitioners will find resonance in the confessions; newcomers will avoid some classic thorns.

Must-Listen Segments

• 09:49–10:50 – Chip/pizza shop analogies about attribution modeling
• 23:30–24:45 – Radical rethink of privacy and tracking
• 43:34–48:52 – Lively exchange on ticketing and request intake systems
• 53:06–53:58 – Lightning round debunking "test everything" dogma

Final Thought

The Analytics Power Hour, in this “I Used to Think… But Not Any More” episode, models exactly the humility, humor, and adaptation that great analytics work demands: admit when you were wrong, learn, and move on—always keep analyzing.