The Analytics Power Hour - Episode #272 Summary

Title: When the Metric is Calculated and Complex with Dan McCarthy
Release Date: May 27, 2025
Hosts: Michael Helbling, Moe Kiss, Tim Wilson
Guest: Professor Dan McCarthy

Introduction

In Episode #272 of The Analytics Power Hour, hosts Michael Helbling, Moe Kiss, and Tim Wilson delve deep into the complexities of calculating essential business metrics, particularly focusing on Customer Acquisition Cost (CAC) and churn rates. Joined by Professor Dan McCarthy, a renowned expert at the intersection of marketing, statistics, and finance, the conversation explores the nuanced challenges companies face in accurately defining and reporting these metrics.

Guest Introduction

Michael Helbling opens the discussion by introducing Dan McCarthy, highlighting his academic and entrepreneurial achievements:

"Professor Dan McCarthy is a leading expert at the intersection of marketing, statistics and finance... he co-founded Zodiac, a predictive analytics firm acquired by Nike in 2018, and later co-founded Theta, specializing in customer-based valuation models." (02:35)

Dan McCarthy prefers to be addressed by his first name, setting a collegial tone for the conversation.

Defining Customer Acquisition Cost (CAC)

The conversation kicks off with a discussion on the challenges of defining and calculating CAC accurately.

Dan McCarthy emphasizes the importance of precise definitions:

"In data and analytics, we don't really talk about that too often, but it really starts to matter as those metrics become more complex and are imbued with meaning intended or not." (00:15)

He explains how varying definitions can lead to significant discrepancies in reported CAC:

"Depending on what you choose to include or exclude, you really can see these really big swings in the numbers because it's such an important metric." (04:00)

Challenges in Metric Definitions

The hosts and Dan discuss how different companies manipulate metric definitions to present favorable outcomes.

Dan McCarthy shares insights into companies that adjust their CAC calculations to appear more attractive to investors:

"There's no universal standard definition for terms even as common as the churn rate... they try to count more things as acquisitions in the denominator." (14:07)

Tim Wilson adds that while some deviations are well-intentioned, others cross into outright fraud:

"But it's not over. Out and out fraud. Right. So we're kind of playing more in the realm of these things feel kind of benign and... it actually does matter." (07:13)

Dan McCarthy cites Netflix's legal issues as an example of the repercussions of ambiguous metric definitions:

"They had been hit by a class action lawsuit over how they had defined Churn... the judge threw out the case because they very transparently provided their definition of churn rate." (06:10)

Impact of Metric Definitions on Business Insights

The discussion highlights how misleading metric definitions can obscure the true health of a business.

Dan McCarthy points out that a decreasing churn rate isn't always a positive sign:

"Sometimes when churn rate moves down, it's a negative signal because it could mean that acquisitions are falling." (09:58)

Michael Helbling advises caution when interpreting metrics reported by companies:

"Note to self, when a software as a service company starts touting their churn rate. Top of the page, dig deeper." (10:18)

Strategies for Accurate Metric Calculation

The guest emphasizes the necessity of cohort analysis to gain meaningful insights into churn and CAC.

Dan McCarthy explains the limitations of aggregate metrics:

"With the churn rate moving down, it's not a sign that anything has actually gotten better. It could be that every single cohort is identical to the previous cohort." (08:34)

He advocates for detailed cohort-level data to understand customer behavior over time.

Challenges in External Reporting

The conversation shifts to the difficulties companies face when reporting metrics to external stakeholders like investors and the SEC.

Dan McCarthy discusses the reluctance of regulatory bodies to standardize metric definitions:

"FASB had a lot of issues because it's very hard to have like a one size fits all rule for any measure." (35:50)

He suggests that informal standards within specific industries, such as SaaS, might be more feasible.

Internal vs. External Metric Definitions

The hosts explore the balance companies must strike between internal optimization and external reporting.

Dan McCarthy emphasizes focusing on the best definitions internally, even if external disclosures use simpler metrics:

"If you're the CEO of the firm, you should really care about like the best definition. That's gonna drive the best possible outcome for the firm." (25:49)

He warns against companies manipulating external metrics at the expense of internal accuracy.

Case Studies and Real-World Examples

Professor McCarthy shares examples of companies like Warby Parker and Toast that have manipulated their CAC definitions to present more favorable metrics.

"Warby Parker... defined CAC to be acquisition related sales and marketing expense divided by the total number of active customers." (13:10)

He critiques these practices, explaining how they can distort the true cost of acquiring customers.

Advancements in CLV Modeling

In the latter part of the episode, Dan McCarthy introduces his company's latest model, CLV Ultra, which enhances the accuracy and automation of Customer Lifetime Value (CLV) predictions.

"Our big push this year has been to really take advantage of the automation factor... it's the closest thing I've ever seen to a free lunch." (51:08)

This advancement allows for more precise financial modeling and supports innovative business applications, such as specialty finance companies leveraging customer data for non-dilutive funding solutions.

Final Thoughts and Recommendations

As the episode wraps up, the hosts share their "last calls," offering listeners additional insights and recommendations.

Tim Wilson references a David Epstein piece on the benefits of stepping outside comfort zones, tying it back to the importance of accurate analytics in fostering business growth and innovation.

Moe Kiss recommends "The Chairman's Lounge" by Joe Aston, highlighting its relevance to understanding CEO incentives and their impact on company metrics.

Michael Helbling issues a cautionary note regarding emotional attachments to AI, urging listeners to maintain a clear perspective on technology's role in analytics.

Conclusion

Episode #272 of The Analytics Power Hour provides a comprehensive exploration of the complexities involved in calculating and defining critical business metrics like CAC and churn rates. With Professor Dan McCarthy's expert insights, listeners gain a deeper understanding of the pitfalls and best practices in metric reporting, both internally and externally. The discussion underscores the necessity of transparency, precise definitions, and the adoption of advanced modeling techniques to ensure that analytics truly drive informed decision-making and long-term business success.

Notable Quotes:

• Michael Helbling:

  "In data and analytics, we don't really talk about that too often, but it really starts to matter as those metrics become more complex and are imbued with meaning intended or not." (00:15)

• Dan McCarthy:

  "There's a lot of companies, public and private, that will basically do whatever they can to make the numbers look as good as possible." (04:00)
  "If you're trying to optimize your business, you probably want more granular measures." (22:53)

• Tim Wilson:

  "They start with the pedal hits the metal... producing things they wouldn't have done had they been purely focused on long term value." (27:37)
  "Revenue is revenue, but people want revenue to be revenue, and you end up in arguments about the right definition." (21:16)

• Moe Kiss:

  "There is a challenge when we're talking about CAC specifically because there's different levels of business maturity." (25:49)
  "There's such an art in naming because people prefer metrics that seem simple like LTV." (49:02)

• Dan McCarthy:

  "If you have metrics that are not internally consistent with other metrics in the filing, you realize there's likely some sort of definitional conflict." (34:28)
  "We need to think about the holistic value of customer across the firm." (43:12)

For more insights and discussions, follow The Analytics Power Hour on LinkedIn or join the conversation on the MeasuredChat Slack Group.