Podcast Summary: The Analytics Power Hour

Episode #293: Tool Selection and the Unhelpfulness of Feature Comparisons
Date: March 17, 2026
Hosts: Michael Helbling, Tim Wilson, Moe Kiss
Guest: Jason Packer (Quantable Analytics, author of Google Analytics Alternatives)

Episode Overview

This episode dives into the challenges of analytics tool selection, exposing the limitations of feature-by-feature comparisons and offering insights into how organizations can more effectively evaluate which tools truly suit their unique use cases. Special guest Jason Packer shares firsthand experience from researching, evaluating, and writing about alternative analytics platforms, emphasizing real-world testing and organizational needs over high-level marketing claims.

Key Discussion Points & Insights

1. The Flaws of Feature Comparisons

[00:16 – 01:59; 38:58 – 41:28]

• The hosts and Jason agree that feature checklist comparisons (“does it do X?”) are quickly outdated and often misleading.
• Jason Packer [39:00]:
  “Feature comparisons are helpful in some ways, but they also are, you know, already outdated by the time you posted your feature comparison list. It's already out of date... their version of X might not be what you really think that you're getting.”
• The consensus: You need to go far deeper than the vendor’s sales materials or table-based comparisons to make an informed decision.

2. Hands-on Testing & Real Data

[02:42 – 11:23; 41:28 – 43:37]

• Jason’s Approach: Actually installing tools, ideally with real data (even if on a trivial or side project), is key:
  “For me, the way to do that is to use it and to use it with real data. Not to watch videos about it, not to be walked through a demo by somebody, but to install it on a website... that's how I learn best.” [04:32]
• Proof-of-concept (POC) implementations surface issues demos never will, like problems handling bot traffic, broken pages, or compliance quirks.
• However, multiple POCs can be resource-intensive, so starting with simplified or pared-down tests (or seeking free tiers) can help.

3. Matching Tool Philosophy to Organization Needs

[12:44 – 20:51; 41:11 – 46:09]

• Organizations often fall into two traps: wanting the “cool” new tool for resume purposes, or defaulting out of familiarity from previous jobs—not necessarily what's best for the business.
• It's crucial to align on stakeholder needs, not just the analytics team’s preferences.
• Jason Packer:
  “A lot of times the tool you have now just isn't implemented correctly. And the new one you get isn't gonna be implemented correctly either.” [16:59]
• Understanding tool philosophy (simplicity vs. extensibility, privacy-minded vs. feature-rich, marketer-first vs. developer-first) is more informative than chasing features.

4. Evaluating Tradeoffs and Total Cost of Ownership

[17:52 – 19:19; 48:49 – 50:07]

• There is no “perfect” tool—trade-offs are inevitable.
• When guiding companies, focus on their real pain points (e.g., compliance, budget, governance) rather than abstract “best practices.”
• Jason and the hosts stress thinking beyond upfront pricing—considering engineering support, onboarding, potential vendor lock-in, platform extensibility, and total cost of ownership.

5. Engaging With Vendors: When & How

[23:08 – 27:13]

• For his book, Jason intentionally did not speak with vendors to maintain objectivity, instead relying on direct product experience.
• In enterprise settings, engaging vendor engineers and support can be invaluable—especially for assessing culture fit, responsiveness, and long-term partnership potential.
• Transparency on both sides, particularly regarding budget constraints and business problems, accelerates decision-making and mutual understanding.

6. The Open Source and Free Tool Fallacy

[30:11 – 32:39]

• Jason and Moe discuss how Google Analytics set an unrealistic expectation that high-quality analytics should be free, distorting the market and undervaluing analytics.
• Open source isn’t “free”—hidden costs in engineering resources, support, and opportunity should be factored in:
  Moe Kiss [31:53]:
  "We’re talking about a solution here that has five full time engineers supporting it. That is not free."

7. Build vs. Buy in the Age of AI

[33:20 – 37:26]

• With the rise of AI and easy prototyping, organizations may be tempted to build rather than buy analytics platforms.
• Jason cautions that base-level builds (especially with AI) lack the maturity, robustness, and support of established tools. However, extending open-source solutions for specific needs can make sense.

Notable Quotes & Memorable Moments

• On vendor marketing and roadmaps:
  Mo Kiss [01:31]: “The only thing you missed is the like, ‘Oh, don’t worry if we can’t do it yet, it’s on our roadmap.’”
• On analyst subjectivity:
  Tim Wilson [12:44]: "You get to the end … and you’re like, all we’ve done is allowed people to dig their heels in further on their preferred tools."
• On matching philosophy to use case:
  Jason Packer [41:11]: “That product philosophy and Outlook is more important than any sort of like feature comparison.”
• On free tools costing more than you think:
  Moe Kiss [31:53]: “Open source or it’s free, it’s not going to cost us anything… that does not mean it’s free.”

Segment Timestamps

• [00:16] – Why feature comparisons fall short
• [02:24] – Introducing Jason Packer and his tool evaluation journey
• [04:08] – Importance of hands-on testing with real data
• [07:31] – Challenges in running POCs with actual data (privacy, resources)
• [10:04] – Negotiations around POCs and evaluating multiple vendors
• [14:16] – Tool preference bias in analytics teams
• [16:44] – Analysts’ familiarity or self-interest driving tool selection
• [17:52] – Trade-offs in tool selection & focusing on business pain points
• [19:38] – Jason's book structure and providing a selection framework
• [23:08] – Why Jason avoided vendor sales teams for his research
• [25:36] – The hidden value of cultural fit and working style with vendors
• [29:03] – The influence of budgets and practical constraints
• [30:11] – The market distortion of Google Analytics’ free model
• [31:53] – The real costs of "free" or open-source tools
• [33:20] – Build vs. buy & the emerging role of AI
• [38:58] – Philosophy vs. Feature: Which matters more?
• [41:11] – How to sense a vendor's underlying philosophy
• [46:09] – Understanding the importance of database “plumbing” and fundamentals
• [48:49] – Jason’s single most important advice: price and total cost of ownership

Key Takeaways

• Go beyond feature lists. Understand vendors’ philosophies, who their target users are, and whether the tool fits both your technical and stakeholder needs.
• POC with real (or representative) data if possible. Demos hide many real-world flaws.
• Budget is important, but so is total cost of ownership. Consider resources required for setup, support, and maintenance, especially for “free” or open-source tools.
• Don’t underestimate analyst and stakeholder bias. Align selections with business needs and stakeholder workflows, not personal preference or career aspirations.
• Be transparent with vendors. Early disclosure of constraints, real needs, and budget helps everyone.
• AI will shift “build vs. buy” conversations, but maturity and support still matter.
• Engage with a vendor’s culture. The right fit goes beyond technology.
• Understand the underlying technology stack. Subtle differences in database or architecture can have big downstream impacts.

Notable Resource Mentioned

• Jason Packer’s "Google Analytics Alternatives" (now 2nd edition): A thorough resource for understanding and evaluating tools beyond surface-level features.
• Measure Chat Slack, Measure Music Channel: Community engagement examples.
• Prism by Ask: AI-enabled SQL and analytics workflow improvement (light sponsor moment, but contextually relevant).

Suggested Action for Listeners

• When evaluating analytics tools, prioritize your business’s real pain points and map them against each vendor’s philosophy and implementation—not just their feature table.
• Factor in total cost (money, people, support, opportunity) rather than being attracted to “free” platforms.
• Don’t hesitate to engage with sales engineers—on your terms—to gauge fit and support, but push for transparency (especially around pricing) early.
• Experiment hands-on, in your context, with real or well-mimicked datasets whenever possible.

Memorable Closing Quotes

• Jason Packer [48:49]: "I might actually say price. ... Price is, like, it's a shortcut to a lot of, you know, putting you in the right area."
• Tim Wilson [19:19]: "Understanding what attributes truly matter and then going deeper, right?"
• Moe Kiss [31:53]: “We’re talking about a solution here that has five full time engineers supporting it. That is not free.”

Further Resources

• Measure Slack: Community for analytics professionals (link in show notes).
• Google Analytics Alternatives (discount code: APH): Jason’s book for deeper dives into selection and methodology.

For analytics professionals navigating a crowded tool landscape, this episode offers practical, hard-won advice against the easy lure of checklists or hype. In the words of the hosts: No matter what vendor you pick, just keep analyzing.