#254: Is Your Use of Benchmarks Above Average? with Eric Sandosham

Summary

Podcast Summary: The Analytics Power Hour — Episode #254: Is Your Use of Benchmarks Above Average? with Eric Sandesham

Introduction

In Episode #254 of The Analytics Power Hour, hosts Michael Helbling, Moe Kiss, Tim Wilson, Val Kroll, and Co-Host Emeritus Jim Cain delve into the pervasive use of benchmarks in business analytics. Joined by guest Eric Sandesham, Founder and Partner at Red and White Consulting Partners, the discussion explores the effectiveness, pitfalls, and alternative perspectives on benchmarking as a metric for business performance.

Setting the Stage: Understanding Benchmarks

The episode opens with host Tim Wilson referencing the iconic "All the children are above average" segment from the radio show A Prairie Home Companion ([00:05]). This sets a thematic foundation for the episode's focus on benchmarks and averages in business analytics.

Guest Introduction: Eric Sandesham ([01:00])

Tim introduces Eric Sandesham, highlighting his extensive experience in business decisioning and operating processes across various industries. Eric's credentials include roles at Red and White Consulting Partners, adjunct faculty positions, and leadership roles at SAS and Citibank Singapore. His recent article on Medium, "The Problem with Benchmarks," serves as the catalyst for the episode's exploration of benchmarking.

Human Nature and the Obsession with Comparisons ([03:40] Val Kroll)

Val Kroll articulates the inherent human tendency to compare, both personally and professionally. She explains that organizations frequently request benchmarks as a "yardstick" to understand their position within the market or relative to competitors. Val emphasizes that while benchmarking provides a reference point, it often serves as an external evaluation rather than a tool for informed decision-making.

Challenges with Benchmark Requests ([04:57] Tim Wilson & [05:05] Eric Sandesham)

Tim shares his personal frustration with frequent benchmarking requests, hinting at the common sentiment of viewing benchmarks as unproductive or overly simplistic. Eric agrees, noting the nuances between benchmarking for startups versus established businesses, particularly in understanding opportunity sizing and strategic positioning.

Distinguishing Benchmarks from Information Signals ([04:57] Val Kroll)

Val introduces the concept of "information signals," differentiating raw data from the actionable insights derived from it. She categorizes benchmarks into "front end" and "back end" signals:

• Front End Benchmarks: Used as inputs for decision-making, helping set targets based on various data points.

• Back End Benchmarks: Serve as evaluative metrics to assess whether an organization is on track post-decision.

Val criticizes the predominant use of back end benchmarks as lazy management, suggesting that organizations rely too heavily on external metrics without integrating internal data collection and analysis.

The Fluid Nature of Benchmarks ([07:24] Eric Sandesham & [07:29] Tim Wilson)

Eric discusses how benchmarks can be misaligned with an organization's unique context, such as differing business strategies or customer bases. Tim echoes this sentiment, expressing skepticism about using benchmarks as definitive measures of success, arguing that they can distract from meaningful internal evaluations.

Blending Benchmarks with Internal Metrics ([10:41] Eric Sandesham & [11:16] Val Kroll)

Eric proposes a hybrid approach, using external benchmarks as one of multiple inputs alongside internal performance data to set realistic and contextually relevant targets. Val cautions that blending benchmarks introduces challenges in attribution due to external noise and varying market conditions, advising that such metrics be used with discretion.

Cultural and Contextual Limitations of Benchmarks ([22:14] Eric Sandesham)

Eric highlights the dangers of applying benchmarks across different cultural contexts, using Net Promoter Score (NPS) as an example. He points out that NPS does not account for cultural variations in feedback, leading to misleading interpretations when applied globally.

Case Study: Net Promoter Score ([24:48] Tim Wilson & [25:12] Val Kroll)

Val counters a question about the validity of NPS by referencing academic research that has debunked its effectiveness as a reliable predictor of business performance. She notes that while NPS is popular for its simplicity, it lacks the diagnostic power of more comprehensive customer satisfaction measures.

Internal Benchmarks vs. Baselines ([32:35] Eric Sandesham & [33:35] Val Kroll)

Val differentiates between benchmarks and baselines:

• Benchmark: A comparative metric against competitors or industry standards.

• Baseline: An internal threshold that represents the minimum performance required for business viability.

Tim expresses frustration with the interchangeable use of internal benchmarks and baselines, arguing that relying solely on internal data can lead to complacency and lack of competitive insight.

The Problem with External Benchmarks ([37:19] Eric Sandesham & [37:43] Val Kroll)

Eric questions the reliability of external benchmarks, suggesting that external data often comes from sources with conflicting incentives, such as large consulting firms that might benefit from client performance appearing below benchmark. Val agrees, emphasizing that external benchmarks often lack the methodological rigor to account for organizational differences, rendering them more noise than signal.

The Importance of Context in Benchmarking ([43:00] Val Kroll & [43:59] Mo Kiss)

Val underscores that benchmarking must be contextual, considering factors like marketing budgets and operational differences. She argues that without understanding the underlying context, benchmarks become meaningless comparisons. Moe adds that sentiment analysis and competitive intelligence should be approached as components of broader market research rather than standalone benchmarks.

Final Thoughts: Rethinking Benchmarking ([50:09] Eric Sandesham & [32:57] Val Kroll)

As the conversation wraps, Eric reflects on the diverse interpretations of "benchmarks" across departments, particularly between finance and analytics teams. Val reinforces that benchmarks should not dictate business strategy but rather inform a spectrum of data points used for strategic decisions.

Last Call: Shared Insights ([51:32] to [64:20])

The episode concludes with a "Last Call" segment where each participant shares personal insights unrelated to benchmarks:

• Val Kroll: Discusses a Medium article on artificial intelligence, highlighting the distinction between problem-solving and problem-finding as key to human intelligence.

• Moe Kiss: Commends Eric's writing style and promotes the upcoming Experimentation Island conference, where the hosts will be speakers.

• Eric Sandesham: Recommends engaging podcast episodes on companies like Costco and Hermes and shares admiration for an Instagram account focused on growth and personal development.

• Tim Wilson: Praises Eric’s weekly posts and endorses Tyler Vigen’s Spurious Correlations, particularly enjoying the humorous academic-style explanations of unrelated metric correlations.

Conclusion

Episode #254 of The Analytics Power Hour presents a critical examination of benchmarking in business analytics. Through insightful dialogue and expert perspectives, the hosts and guest Eric Sandesham challenge the conventional reliance on benchmarks, advocating for a more nuanced, contextual, and internally informed approach to measuring business performance. The discussion emphasizes the importance of understanding the limitations and potential biases inherent in benchmarking practices, encouraging listeners to seek a balanced and informed methodology in their analytical endeavors.

Notable Quotes

• Tim Wilson ([00:08]): "Analytics topics covered conversationally and sometimes with explicit language."

• Val Kroll ([03:40]): "It's such a built-in phenomenon as a human species to always compare while we're growing up."

• Eric Sandesham ([10:41]): "What if you are using those external benchmarks as an input to help you set your own target as one of, say, many inputs?"

• Tim Wilson ([09:10]): "You get somebody… and that just smacks poor business management thinking."

• Val Kroll ([24:51]): "Net Promoter Score… has been debunked academically because it doesn't hold up to scratch."

• Eric Sandesham ([22:14]): "iPhone users versus Android users… customer lifetime value is different."

• Val Kroll ([37:43]): "If you can fiddle a number to make the client happy, that's not going to be useful."

Key Takeaways

1. Benchmarking as Comparison: Benchmarks are often used as external comparison metrics but can be misleading without considering contextual differences among organizations.

2. Front End vs. Back End Benchmarks: Benchmarks serve different purposes depending on whether they are used to inform decisions (front end) or evaluate performance post-decision (back end).

3. Limitations and Biases: External benchmarks may carry inherent biases and lack the methodological rigor needed for accurate comparisons, making them unreliable as sole performance indicators.

4. Internal Metrics and Baselines: Establishing internal baselines is crucial for setting realistic performance thresholds, independent of external benchmarks.

5. Cultural and Contextual Sensitivity: Benchmarking metrics like NPS can falter when applied across different cultural contexts, leading to inaccurate interpretations.

6. Holistic Approach to Data: Effective decision-making should integrate multiple data points, including internal performance metrics and contextual market research, rather than relying solely on benchmarks.

Recommendations for Listeners

• Critical Evaluation: Approach benchmarking requests with a critical eye, questioning the relevance and applicability of external benchmarks to your specific organizational context.

• Integrate Internal Data: Prioritize internal performance data and establish baselines that reflect your unique business environment and strategic goals.

• Contextual Analysis: Always consider the broader context—such as cultural differences and market conditions—when interpreting benchmark data.

• Educate Stakeholders: Engage in conversations with executives and stakeholders to clarify the purpose and limitations of benchmarking, fostering a more informed and strategic use of performance metrics.

Further Engagement

Listeners are encouraged to engage with the hosts and guest through various platforms:

• Social Media: Follow @nalyticshour on Twitter.

• Website: Visit AnalyticsHour.IO for more resources and to submit topics or guest suggestions.

• LinkedIn Group & Slack: Join the Measure Slack group for ongoing discussions and community support.

Upcoming Events

• Experimentation Island Conference: Scheduled for February 26-28, the inaugural conference promises insightful sessions on benchmarking and optimization, featuring speakers from the hosts' team.

Final Note

As the hosts humorously riff on the episode's content in the closing remarks, the underlying message remains clear: benchmarks, while popular, should be navigated thoughtfully and supplemented with robust internal analytics to truly drive business success.

#KeepAnalyzing

Transcript

[Val Kroll]

Welcome to the Analytics Power Hour.

[Tim]

Analytics topics covered conversationally and sometimes with explicit language. Hi everyone. Welcome to the Analytics Power Hour, where all the data is strong, all the models are good looking, and all the KPIs are above average. This is episode number 254 and listeners to this show who are also public radio nerds of a certain age will absolutely get that reference. I don't even know if my co host will get that reference. Val Kroll, did you get my public radio deep cut there?

[Mo]

I'm sorry to disappoint him, but absolutely not.

[Tim]

Okay, your dad totally would have, so this will be a conversation to have with him afterwards. And Mo, like you're in Australia. I don't even know if the ABC or any other service ever carried Prairie Home Companion. Do you have any idea what I'm talking about?

[Eric Sandesham]

I am totally and utterly baffled right now.

[Tim]

Oh, right.

[Mo]

Great place to start.

[Tim]

That's how things go for me in most of my social interactions. So I'm Tim and I just did a little benchmarking right there on my Garrison Keillor knowledge relative to my co hosts. So for listeners who don't get it, I'm referencing a radio show that ran for years in the States that included a segment called the News From Lake Wobegon, which was a fictitious town in Minnesota, and the segment always ended by the host Garrison Keillor noting that in Lake Wobegon all the women are strong, all the men are good looking, and all the children are above average with those exact beats. So that last bit is sort of a lead in to this episode because we're going to be talking about averages, specifically benchmarks, this oft requested comparison metric that we get so often from our business partners. Personally, these sorts of requests tend to trigger me to curse profusely. For now though, I'm just going to introduce our guest who wrote a pretty thoughtful article on the subject on Medium as part of his first year of pinning one post a week on the platform, which is impressive. Eric Sandesham is a Founder and Partner at Red and White Consulting Partners, where he works with companies across a range of industries to help them improve their business decisioning and operating processes. Eric is also on the adjunct faculty at Nanyang Technological University, Singapore Management University, and the Wealth Management Institute. He was previously the Customer Intelligence Practice Lead for North Asia for sas, and before that was the Managing Director Head of Decision Management at Asia Pacific Consumer bank at Citibank Singapore, and today he is our guest. So welcome to the show, Eric.

[Val Kroll]

Thank you very much. Thank You. Thank you for having me.

[Tim]

Tim, I should ask you, have you ever heard of Prairie Home Companion or.

[Val Kroll]

I've heard. I've heard that phrase.

[Tim]

Okay.

[Val Kroll]

Yeah. But I've not heard the radio. The radio. The radio broadcast, obviously. Yeah.

[Tim]

Or you're at least being. I mean, I've seen Garrison Keillor live, so I'm telling you there I'm going to get some slack messages. People saying, I can't believe they hadn't heard a prayer Home Companion. But I will, I will not belabor that any further. Trust me, it killed in a certain group. So, Eric, I sort of noted in the introduction that we, or actually Val, came across you because of a post that you wrote on Medium, and the post was titled the Problem with Benchmarks and it had a subtitle which was why are we obsessed with comparisons? So maybe we can start there, Eric. Why are we obsessed with comparisons?

[Val Kroll]

I think it's such a built in phenomenon as a human species to always compare while we're growing up. I'm sure our parents always tell us, don't compare with your neighbors, don't compare with your friends and colleagues, just compare with yourself as long as you're making that onward progress. But I don't think any of us really stick to that. It's just so natural. When you step into a room, you get into a new place of work or in anything that you do, you're trying to size yourself in relation to someone else. And I think maybe it's trying to understand our place in the larger scheme of things. And this carries over into the business world. The first thing, typically most organizations ask for. And as you mentioned, I run my own consulting practice at the very start of many of the consulting practices, the engagements, the clients would ask, can you help me with some benchmarks? I'm trying to get some information and reference points and all of that. Once we get there, we can go into the deeper stuff. Right. But it seems always top of mind for them to want to have a sense of almost like a yardstick. Right. Or a placeholder to know where they are on that map. Yeah.

[Tim]

Mo, does your team get hit with requests for tracking down benchmarks, creating benchmarks, internal, external?

[Eric Sandesham]

Yeah, I have some pretty strong thoughts that are going to come out in today's episode, I suppose. Yeah. Pretty often. And one of the points that I, I think is quite interesting. I don't know. And I'm, I'm really gonna. Yeah. Interested to see how the conversation goes, Eric, because I think it is maybe different when you're Like a startup or in an earlier stage of a business, and particularly like trying to understand opportunity sizing. So I'm curious to, to kind of get your perspective on that of like where there are useful comparisons to make.

[Val Kroll]

Okay, so I think in the article that I wrote, I'm a big fan. Let me take a step back. I'm a big fan of the way we look at data in terms of information signals as opposed to just data as data. I constantly ask myself when I look at any piece of data or any report, what is the information signal or signals contained within there? And so the same thing I would apply to benchmarks. What kind of information signal is the client looking for when they make such a request for a benchmark? And to keep things simple, I think of it as both a front and back end sort of information signal. You're either thinking of benchmarks as an input to a decision, you've got certain uncertainties about your decision making process and say, well, if I know some stuff that I didn't know before, I would make a little bit of a better decision, or you are looking at it on the back end where you say, look, I've already taken the decision, but I don't know whether I'm on track and can the benchmark therefore give me that sense of my place and whether I'm still keeping to the path that I intended to go on too. And so simplistically, front end, back end. And I think most organizations when they look at benchmarks tend to look at it from a back end process as an evaluation information signal. And therein lies the problem. Because is it the right way to evaluate whether you're on the path? Is it the right way to evaluate your actions and your strategy in comparison to others who may or may not be doing the same thing because the.

[Eric Sandesham]

Business strategy is different or the customer set is different or Exactly.

[Tim]

Well, and I think you had that front end versus back end because the front end. And that was a little bit of a, I thought, ah, I'm so used to just like raging against them because I feel like I'm generally being asked, I am way less kind as to saying it's human nature. And I attribute it more to, it's a way to duck out of saying, well, I don't need to figure out what I'm expecting to achieve. Just do the thing and then find me a benchmark to compare it to. And if I'm above the benchmark, I'll say yay. So that falls under that back end, that front end. Part was actually pretty interesting. Like you called out that like salary benchmarks, you know, if you're an HR department and trying to figure out where are we relative to the market. And I've worked at companies that have said we want to pay at market because we think the quality of the work and the living is way better. So we have the secondary benefits are worthwhile or hey, it sucks to work here. We've got to, we've got to pay above benchmarks. So that to me, I was like, oh, okay. So not that those are perfect like that. There's still all the noisiness in trying to get those sorts of benchmarks. But you brought up pricing as another one to say, if I'm selling something, I need to figure out what's kind of the normal sell rate because I need to think is what I'm offering higher value or lower value and adjust accordingly. So that I thought was actually pretty useful. It's that back end just feels like lazy to me.

[Val Kroll]

Yeah, I think I like the point you're making about being lazy in the sense that, you know, get somebody. Well, the consultant is here anyway. Since we're going to pay the consultant, why don't we get them to do all the measurement for us and just doing the external measurement? I sort of absolve myself of even doing any internal metric collection. Everything is just going to be evaluated by something outside of the organization. And that just smacks poor business management thinking. I mean, if you want to try and complement with some external with internal, I think great. But very often many of these companies that ask for benchmarks are just looking for the external evaluation and at the end of it, whether you're up, you're down, so what, you know.

[Tim]

It'S time.

[Val Kroll]

To step away from the show For.

[Tim]

A quick word about Piwick Pro, Tim, tell us about it. Well, Piwick Pro has really exploded in popularity and keeps adding new functionality. They sure have. They've got an easy to use interface, a full set of features with capabilities like custom reports, enhanced e commerce tracking, and a customer data platform. We love running Piwick Pro's free plan on the podcast website, but they also have a payment plan that adds scale and some additional features. Yeah, head over to Piwick Pro and check them out for yourself. You can get started with their free plan. That's Piwick Pro. And now let's get back to the show.

[Eric Sandesham]

Okay, so what if we blend them? This is not where I thought I was gonna go. But what if you are using those external benchmarks as an input to help you set your own target as one of, say, many inputs. Like one of your other inputs might be previous performance or like another department's performance or something like that. And then this is one of those inputs you're choosing to help you set whatever your target is going to be. Like, what is. What's the view of that? Yeah, view of that.

[Val Kroll]

No, I think that's valid, right? To say, look, can it be an additional supplementary metric? Right. The challenge would be when you're using external metrics, the problem will be on attribution. And so whether you can either make that direct connection and attribute the change, let's say, in your market share, or a satisfaction score, attribute it to a particular action or set of actions that you took. And typically the answer would be, it's difficult because there's so much noise. There's just so much things happening, even both internally, and then the market is doing its own thing. So at best, you can just can sort of hand wave and say, looks like it's affecting it, but you can't say it with any certainty. Right. So even if you're supplementing, you would still have to take it with a pinch of salt.

[Mo]

And I think that that's a really good perspective mode, like the blending and using that as potentially an anchor to think about. Like, okay, so here's what's happening. But we can set our own target, whether it's below or above. But I think what you were just calling out, Eric, is an important consideration because the strategies are different, because the consumers are different, because, you know, you think you're playing in this space because, like, that's where you're strongest. Right. So there's all that context where, like, that kind of devalues that number a little bit. Even if you were to, like, average it out with the, what the target setting activity from the voices in the room. Right. Just because you don't have that context. And so that actually might undercut some of the value of the discussion that's happening internally about what success looks like. Right?

[Val Kroll]

Yeah, that's true. And again, you can define the market or the reference point in so many different ways, which many organizations do, to always come out smelling roses, as they say. Right. I'm benchmarking myself to just one other person. Well, you know, somebody's going to win, and that's part of the challenge.

[Mo]

Also, one other thing, though, I think, like, the sincere, the. The information that comes along with the request, if you're trying to, like, break that down, like, what is someone actually craving? I Like, the way you were talking about that earlier, too, Eric. I think sometimes people are worried that if we were to set a target ourselves, would that not be aggressive enough? Like, if maybe we're setting the bar too low or we're not thinking about, like, you know, a larger piece of a puzzle. And so they're doing it as a way to make sure that they're being aggressive and that they're not going to be outpaced by just setting the targets that are internally. Again, if you get to that conversation, Tim's. If you only could see Tim's faces.

[Eric Sandesham]

Tim is calling absolute bullshit on this. I can see it in his face.

[Mo]

I mean, I'm talking about, like, if you get to a convert, if you're already getting to a place where your organization is comfortable with talking about those targets. And. And so this is a special beast, right, because we. I totally agree with you wholeheartedly, Tim, about laziness. I think that that's like a shorthand way of saying, like, you know, some resistance against the. The finality of setting, you know, drawing a line in the sand. But I do think that sometimes it comes from a good place of making sure that we're not setting something and it's, you know, just too easy of a target or, you know, then we achieve it and then what.

[Tim]

But you're doing something to achieve a result. And if you're saying, this is going to cost me something in time and money, and I would be happy if I spent this amount of time and money and achieved this result, to me, that's the core of what a target is. And if you said, well, yeah, yeah, but we compared that target to some super unattainable, messy AF benchmark, and that's not aggressive enough. Like, that's. To me, that's, like, super distracting from the conversation. If I say, if we put $1,000 into this and we get $8,000 back, are we cool? And everybody's like, sure. You're like, well, I actually sandbagged. If I found these benchmarks, which, by the way, don't really exist, then I should have set it at 10,000. Like, that's.

[Mo]

I'm not saying it's a fruitful exercise, especially when we get into conversations like, well, Netflix runs this many tests a year, so I think we should. Okay, great. Because you're definitely built to run like Netflix. So I'm not saying it's a fruitful exercise. I'm just saying that's the place where that comes from sometimes.

[Eric Sandesham]

So can I just touch on There was something as I was digesting Eric's blog post and I did actually try to figure out which one it was, but I can't remember off the top of my head. And I'm hoping Tim, who's smarter than me, will remember. But there is this bias that people have to be like, what. What previously has been done so I can use that as a frame of reference. Like we're saying, like, benchmarks aren't great use this way. But like, this is human intuition of like, what previous experience or what have other companies done? What, what is the. The data quote, unquote. Whether the data is right or not or useful or not is a different point. But like, we're talking about a human bias to seek what previous experience can I rely upon to understand what good is? And so like, we're talking about it like it's, you know, the worst thing ever. But it's like this is our natural intuition as humans.

[Tim]

I mean, my, my quick take is that, well, I would rather that I just sit down and just think about what, everything that, what I've learned or seen or known before and what I'm doing and what I expect to get out and start with that and then say, if I. I'm going to be way, way more emotionally tied to and invested in that than if I just had the analyst or the consultant go pull me a number, right? Because if the consultant comes and gets a benchmark and then we wildly miss it, then it's like, oh, you must have pulled a bad benchmark. I've circumvented any ability to say, did this thing do what I expected it to do or not? If I pull that myself now I have information of what I did, and now when I'm considering the next thing, I can say, well, I got to lower my target a ton. Which hopefully may make me say, is it worth doing or not? But I don't know. What do you think, Eric?

[Val Kroll]

Actually, this is cutting it at different, I feel, at different perspectives. So the word benchmark, I think is a loaded word because it can mean many different things to many different people. As you're talking, Tim, you say someone has this external reference and you're using it as target. For example, am I doing well enough to achieve a particular reference target? Yeah, you're right, that's a benchmark, but it's a different kind of benchmark from, say, if I'm looking at relative market share, where I'm looking at a ranking, or even if pay salary, it's not a target. We Use for reference of how we want to pay. But you want to know what's your relative standing, what's your relative rank with your competitor. I think there are mo raised a good point to say there are stuff where we aim to achieve, for example in certain practices where there's a long duration or history of maturity to it and it's become almost commoditized. And if you are a startup and then you're saying look, if I'm going to do this then I actually have to get to the similar state run rate of whatever everyone else is doing. So for example, I come from the retail banking side. If you take a credit card business, interesting benchmark would be CPA cost per acquisition and you say look, everybody has to acquire a new credit card customer. We can't have very widely divergent cost of acquisition. At some point, if you run this well and you're matured, logically all of us will begin to converge around a very narrow range of value. And if you say is my business healthy? Am I doing all that I can to take out the waste and being effective in my targeting, then logically I should be within that narrow range of cpa. And if I'm not, then there is something I'm missing perhaps. And it can be an interesting way as a diagnostic measure to say what am I not seeing right, what am I not getting right in my process? Because if everyone's converging then logically I should be.

[Tim]

But that, I mean that's, that gets too, I mean in, in the banking sector I feel like I've had, when I've had questions or people have asked me for what's a benchmark E commerce conversion rate for, you know, retailers. And I, I was like, I asked them, I'm like, are you, have you shared what yours is like publicly? Like have you shared that with a trade? And they're like oh my God, no, like I wouldn't share that. It's like, well but you think your competitors are sharing that. And I even think like in banking or a cost per acquisition, you know, if you're looking at your customer, lifetime value for different customers and financial services varies hugely. So like an average cpa, when, if you're selling to a super high end or you've got another product that's selling to super low end, like even like within your company, you may say well we don't look for the same CPA for the different products because these are much more valuable and these are much lower value. But I mean it just, it feels like Such a slippery slope that you go down to try to get those. No, I mean it happens with salary information too. When you say it works for very specifically clearly defined roles where there are regional differences and there's enough scale and there's enough data collected. But I think we've all watched companies that HR struggles when they say, well, the service we subscribe to for our salary benchmarking doesn't have, you know, an analytics engineer role. So we're just going to name our people data scientists and then we'll compare to that. I mean like, it's a bizarre. So that I agree is like a really, it's a really good use case, but it still runs into this mode of like there's this idea that the benchmark is much cleaner with, has much less uncertainty around it then it, then it than it does. And so like, like, where does it come in?

[Eric Sandesham]

And can I just add to that, which is hysterical because I feel like I'm now arguing against myself. But like a really good example, right, is like iPhone users versus Android users. You might be like, oh, pull a benchmark report on mobile usage. And it's like, like I know from looking at those the customer set is very different. The same is true of like Amex and Visa. Very different, like customer lifetime value, that sort of thing. But you're like, oh, look at credit card usage. And you're like totally different audiences and yeah, lifetime value. So I think that is where it can get dangerous. But hypothetical situation for you, which I love, let's say you are, I don't know, you have an iPhone app, you're trying to decide if you should invest in an Android app and you don't have any historical data on the Android app because it doesn't exist. So like you need to make a decision about. And this is probably getting in the front end side of Eric's thinking, you need to make a decision about whether you should invest in Android. Is the comparison to the iPhone usage that you have internally correct? Probably not, because as I just said, different customer group. So like in that situation I can see that some type of benchmarking from competitors might be useful as one of the inputs into helping you make a decision.

[Val Kroll]

But I think in that, in that, in that perspective, if you're looking for information to sort of bushwhack your way forward, then that's more, more like research. Yeah, yeah, that's a bit of market research and benchmark for the purpose of saying, you know, well, it is collecting benchmarks, but actually you're collecting Information signals for the, for the purpose of market research. Right. And you're completely right. It's on the front end because you're trying to decide which path I should to take to forge ahead. Right?

[Tim]

Yeah, yeah.

[Val Kroll]

But often my struggle in the consulting is the organizations or the clients asking for benchmarks actually are not clear exactly what they would do with it. The so what question, if you ask them, of course they get a little bit offended and defensive. Right. We had a client stop asking. We just want our benchmarks. But the reality is that I don't think they really know what they would do with it. But it's a great piece of information to bring up to senior management and to the boards and all that. You know, this is various benchmarks and where we stand, it has its appeal, obviously. Right. But I don't see how many people use it to take better decisions.

[Eric Sandesham]

Can I ask a controversial question?

[Val Kroll]

Sure.

[Eric Sandesham]

I don't know if you're familiar with Net Promoter Score. I mean, that is the ultimate benchmark, right. And there are lots of research from consulting, you know, that says that it's very closely tied to the revenue performance of a company. What are your thoughts on using that as a benchmark?

[Val Kroll]

Okay, so in the academic space, actually the Net Promoter Score, nps, right, for short, has been debunked. Right. In case people are not aware, actually it's been debunked, academically debunked because it doesn't hold up to scratch. It looks like a great, nice shorthand. And the reality, again, a lot of stuff that gets onto the business world intuitively feels familiar, it's easy to run with and sometimes takes on a life of its own. And then the actual research or validation happens much later and then it's sort of never see the light of day. But academically, actually the paper has been debunked. 1, 2. They found that actually it doesn't give any sharp diagnostic or measure versus how organizations used to do it with their customer satisfaction, where they had multiple questions and you could slice and dice. And there was no lift against the previous methods that people employed. But it was a nice shorthand. But of course, with things that are shorthand, you introduce noise. Then obviously, I mean, when I first saw it and encountered it in my corporate life in Citibank in Asia, I came out of the US with Bain and working with the professors, it was like really you benchmark to say anybody above 7 or 8 is good, anything below that, shouldn't 5 be the average? But the average is much higher than the Median point, we think. But in Asia, everyone's conservative. No one's going to tell you you're good. Everyone, I mean, in Asia we sort of understand this. Never, never give good, you know, good, good feedback. Always criticize, right? Because if people's head will get stolen and they all feel great about themselves, in Asia we take the opposite stance. Everyone's not good enough. Right? So even customers will never, never give you that good feedback.

[Eric Sandesham]

And that's such a good point actually about benchmarks being quite dangerous. Is that like it doesn't account for cultural differences or many of the other differences, right? Like cultural just being one of them. When you, when you take an average like that and try and apply it broadly.

[Mo]

I have agree. NPS has a special place in my heart. We, when I was in market research at the beginning of my career, there was a cable provider that we did a lot of customer satisfaction and also transaction based satisfaction. So after you had to interact with customer service and when NPS came out and everyone was reading the book and everyone's talking about how this is like miracle care, they embarked on an NPS study where it was only five questions and it was just getting at NPS for all 70 plus divisions and they ran it on a monthly basis. But if you think about the also like the SKUs of the fit, right? Like it started with like airlines and hospitality where there was a lot of choice and it was a lot easier, like lower switching costs. But we're talking about how likely are you to recommend to a friend or colleague, your cable provider, like first of all, whoever is having that conversation at a party, like, that's curious behavior. But also a lot of, especially at the time, this is like, you know, 2010, you, you actually couldn't switch to all the different providers and they were benchmarking themselves against Dish Network and that was one of our favorites because they were like, oh, people like Dish better than us. And it's like they had to think that your service was so terrible that they would pay to have a, you know, 20 foot dish installed on their house to try to circumvent the service you could provide them. Right? So it just ended up being this. How do we explain the volatility month over month across these 70 plus divisions? It was like the craziest, wildest ride. And I remember our in house statistician would have to like breathe into a paper bag when we get on the calls of the client to try to like, well, how do you explain why this one's up in this one's down. And it's like. Because it means nothing. It means nothing. It was just like the wildest experience. I'll just never forget.

[Tim]

Well, well, the volatility. I mean it kind of goes to the, the bane of analytics in general when a metric that's noisy, if it's a noisy metric and it moves and you look at it and it goes up and people are like, why did it go up? I'm like, noise. It went down. Why did it go down? Noise. I've twice worked at agencies, mid sized agencies that had. We're doing NPS, you know, B2B small sample. You're taking a 10 point scale and chunking it into three buckets and then doing subtractive math on it. I mean it's just like it's taking this one thing and making it so crude. And then they were doing it with a small sample size and it was just every quarter some percentage of the clients would get hit up with it. Some smaller percentage would respond to it. It was just a noise generator. But by golly, if it went up, we'd hear about it. And if it was down low, nobody talked about it. And, and, but there would be that thrown out that a NPS above. I don't even know what the number is.

[Mo]

United Airlines got a 70, you know. Yeah. What does that mean? That's not a helpful benchmark.

[Tim]

But, but, but on that like asking a question, especially in a, in a D2C context, like asking how likely are you to recommend? I mean like, oh, you're selling to like at scale to consumers, like it seems like you could get volume that there would be. It's a. Seems like a reasonable question. Would you recommend us especially if we're in a growth area. How many of our clients, customers would say they would recommend us? That actually feels like in certain contexts, depending on where our strategy is, is a fair question to ask. It's when it gets jumped to now subtract the detractors and do this and to compare yourself to a bench. All of a sudden now it feels like it's gone into the. Nope. I'm just trying to, you know, get a number that makes me sound good.

[Eric Sandesham]

Okay, maybe now that we have had our NPS deviation, can we please talk more about I guess the front end benchmarking side because. Okay, can I throw another scenario out there? Let's say you're looking at. I'm trying to, I'm like thinking about this on the fly marketing budget, right? And you're going, you're trying to basically Determine which markets are worth investing in. You've got a finite budget. You can't go into all of the markets. So maybe you look at like total addressable market. You look at tam, you look at gdp like you look at a bunch of. You might also use your own internal, like monetization rate or something like that. Like, it seems like in that scenario, using benchmarks is appropriate. But I noted, Eric, in your article you said we should never start with comparisons unless they help shape our decision inputs. Most don't. So is this scenario shaping our decision inputs?

[Val Kroll]

I would say yes. But I would also then challenge, are these kinds of information really benchmarks? Because again, the word is so loose, right? So you say, give me GDP of the various countries or options that I want to go invest in. Is that a benchmark?

[Eric Sandesham]

So you're saying no, or is that just market research? Okay, so we're saying it's market research.

[Mo]

It's market research, primary and secondary research. It feels like if just because it's desk research, just because you're not going to your customers or prospects, that's still like research input. I would agree with that.

[Val Kroll]

Yeah.

[Eric Sandesham]

Okay, so then let's rewind to the thing that I should have asked right at the very start. How would you define a benchmark then, Eric?

[Val Kroll]

Okay, yeah. So for me, a benchmark means I, strictly speaking, in the space that I sort of rant about, it's the relative difference to a competitor or to a space that I operate with. Right. So if I have a way to compare myself with somebody as opposed to, say, is the market a good market or bad market? That's not necessarily about me comparing with somebody. Right. Then to meet a benchmark, if it's competitive comparison, you know, has some facet of competitive comparison, would, would be a benchmark for me.

[Mo]

And how is that different from a baseline? Because I think that that's, that would be good to tease a point.

[Val Kroll]

Okay. Okay. So, so a baseline for me is a sort of a hurdle, a hurdle that you want to get over. So if you are starting out and you're, you know, building some capability, the baseline should be something that you try to get over again, like the cpa. So if you think of cpa, I think CPA has this, this, this sort of, you know, two sides to it. Right? Because at some point, if, if, if you're going to compete like a, a mass credit card with everyone, then, you know, you, you, when you first launch your product, obviously the CPA is going to be high because you're trying to Win over market share, trying to get your brand out there, awareness and all of that. But once you've got a matured business, logically you need to get over some kind of baseline cpa. Now, that baseline CPA may be very different from a benchmark comparison CPA where you have a variety of different competitors at various levels, again, different segments they go after, and so on, so forth. But a good operating business would say that if I can't get a CPA of X dollars, then then I'm not going to be running a profitable business regardless. Right? And I think to me that baseline is about some kind of minimum hurdle that you want to get over that the business makes sense.

[Tim]

This is another axe I have to grind is like, I feel like the other thing I will hear in the CPA example, it would say, well, what CPA do we need? And it'd be great if there was like math done to say we've got to at least, you know, the CPA has to be below this or we're not going to be profitable. I feel like I run into more often, we've never done this before. So let's just run it out there and let's get a, let's get a baseline for our CPA and then we'll know going forward what it needs to be. Which also sends me kind of around the bend because the way you just articulated is saying, no, you're, you're setting a baseline as opposed to I'm just going to do whatever and kind of do a good job and gather some data, because that's a can that can be kicked down the road again and again and again. There's always an excuse to say, well, I don't have a baseline. I don't have a historical internal data. So I kind of tend to think of like an internal benchmark and a baseline as being kind of comparable. But I get irritated with them as well because it's also like, you can't possibly expect me to set a target for what good is. And because I've never done this before and I'm like, bullshit. I can't, you know, but so wait.

[Eric Sandesham]

Sorry, Tim, are you saying that an internal benchmark and a baseline are kind of interchangeable? Am I following that?

[Tim]

I feel like that's how I see it used, intend to use it. I don't, I don't know that that's a hill I'm gonna die on.

[Mo]

I like it like that because I do think because a lot of times inside an organization you do have the context that you don't have when it's an external benchmark. Right. Like oh well this one had like half the marketing budget. So you have to take that in consideration. It's kind of like you have the use with caution, kind of like the baked in assumptions or the things that are kind of really different about that point of comparison so you understand how valuable it can be to helping you set context. And so that is one of the reasons why I think about that too. Just because it gives you, you do have that background, you can source the information again to assess how helpful it is.

[Eric Sandesham]

Couldn't you do the same for external benchmarks too? I get that maybe you don't have as much context, but you could still have like use with caution warnings of like this is what we do know about how they were created or whatever. Or is it like you just feel like it's way too black boxy? Oh, everyone's shaking their head. So I'm going to assume I'm hell super wrong on this.

[Val Kroll]

Yeah. So for me, I think most information, most data are equivocal in the sense that you know as equivocal meaning we have multiple interpretations and often conflicting interpretations. And that's the issue, right? When it goes up, someone can say oh you've done a good job or it's just noise or vice versa. I think if we start and say, look, whatever the benchmark that exists, how much equivocality does this benchmark have? And I think if we hand to our heart and we're honest, look, actually there's quite a lot of equivocality in terms of the benchmark, then is it going to be useful? Because ultimately, even as an evaluation metrics, we are biased in the sense that we reward ourselves for all the successes, whether it's on us or not. And then we'll try to find excuses and reasons for why we fail. And if you can make a valid reason because the metric is equivocal, then does it really help you chug along? Right.

[Eric Sandesham]

Such a good point.

[Tim]

And that is like if you see it and you exceed it, you're like, look, this magical thing, we did better. And if you see it and you did worse, then you say it must be garbage.

[Val Kroll]

The market worked against me.

[Mo]

Right.

[Tim]

I mean we had VAL and I had the same client, this was a few years ago that had an agency that had multiple clients. But when they would their their media results and they would less cpa, just the nature of where they were, they would look at cost per click or cpm, which is super, super common. And they would say they were constantly reporting that they were, you know, beat benchmark. And they would say, and guess what? Because we have so much data, you're beating benchmark for your sector. And I'm like. And the client would just take it and say, look, this campaign was great. Our CPM was below benchmark or our CPC was below benchmark. And what's like yeah, but that's such a noisy thing. Like no, no, no. They told us that the data they were using was for totally apples to apples, which is all the kids are above average.

[Val Kroll]

Just sorry to interrupt and jump in. So this client that says that my CPM is below benchmark, look how well I've done. You can also flip that narrative and simply say, did we under invest? Did we leave money on the table? Because if we were at benchmark, couldn't we make more?

[Eric Sandesham]

Can I flip this? So in my mind and again, people might violently challenge me on this. There tend to be kind of like two trains of thought I have found when you're working with executives. One does tend to be the like, how are we doing against our competitors? And I do find then there are also the execs that are like, I don't care what our competitors are doing, we're running our own race. How are we comparing year on year or like to the last time we did like very much about internal comparisons. If you have got the one that is very focused on like how we doing against our competitors. I feel you feel like this benchmarking discussion is something you would need to bring up. How do you think you do that in a constructive way that would get them I guess to see the like, I don't want to say like the errors of their way because that sounds super patronizing. But like how do you start to educate them about this?

[Val Kroll]

I think in business you, you definitely have to have competitive information, right? Whether it is an the form of benchmark or not. I mean business is not a one man race, right? You're obviously competing in a space with others. And so to say that I'm just going to isolate myself and just look at internal metrics and then yay, I'm successful or not, I think that's not wise and definitely not realistic. But to run a business entirely based on competitor evaluation and where I am at the point in time, it's also meaningless because then you don't have a mind of your own. And making a decision whether I want to stick to it or not, I think it's really the collection of information that you would use. So if you're saying, look, I want some competitive benchmark, then it is because I have some kind of evaluation or decision uncertainty that I can fail in with that. Right. Recognizing also that the minute I go out of the organization with external information, then there is a lot more noise. And I think, I don't think people realize that because they are thinking internal matrix and external matrix. Yeah, they all have variants. They're not the same kind of variance. The internal matrix, in many instances, you can control the variance even if there's noise. I can always identify isolated because I know something about my process. But with the external one, you don't even know the nature of the noise, let alone wanting to try and control that. Right.

[Mo]

And also just say, like, there's, there is a lot of value in getting competitor information, like in context of a decision you're going to make. And so one area that I've seen a lot of clients do it, especially my bias coming from market research, is understanding sentiment or like, attitudes. And so sometimes shifting away even from like NPS to understand, like word of mouth, like, if that's what it's really trying to get at, then let's ask some questions about that or, you know, how likely are you to X, Y, Z behaviors? And I think some of those are helpful to capture against competitors, too. And that can be informative of where you play or how closely you are delivering on your value proposition or differentiation from key competitors. But again, I don't necessarily consider those benchmarks because if you're still saying we're going to have a separate conversation to evaluate our own performance, the choices we're making, like that can still just kind of be more on the input side. But Eric, you can let me know if I, if I misinterpreted that.

[Val Kroll]

I would agree. So if you're saying, look, I need this sentiment analysis. Right. Then of course the challenge would be who does that best and how do they do it so that it's comparable. And they've sort of normalized the noise in it, I think where, you know, the rent is as sort of, you know, startup consulting and all of that. It's strange that when I talk to clients and they know that I'm a boutique consulting business and say, well, you know, can you get me benchmark? Well, yeah, I mean, I've consulted for a range of some of the clients, but I'm not a McKinsey. Right. I'm not an Accenture or a Deloitte where you say, you work with everyone and you sort of have steam your ins and outs in those businesses and approaching a small startup for external benchmark, even though you can say, yeah, maybe they're prepared to do it, not because they need your business and all of that, but they don't really have that kind of methodology that would stabilize the noise.

[Mo]

Yeah, that's a good way to put it.

[Val Kroll]

And so you can get a number that ultimately and again, you know, you can fiddle a number to make the client happy and that's not going to be useful.

[Tim]

Well, and that's a, I mean like you take the big, the large, large scale consultancies that say we have a massive customer database and we therefore have access and we are going to obfuscate it and develop benchmarks for you. That tends to be, you know, that's what Boston Consulting Group or McKinsey or Deloitte is trying to sell you. Like, so you're, you're the organizations that, the metrics that they're going to be the tightest and cleanest on gathering their benchmarks just happen to be for metrics that they and their services they, you know, say, well, they will help you with. Right. Like, so there's, there's a little bit of a fox in the hen house of that they may have, they may factually be accurate and they're probably behaving pretty well like they're not out there being like malicious. But they do have sort of perverse incentives to have the new client or the prospect be performing below the benchmark because that's how they're going to get paid. So even like considering who's doing the aggregation of the, like the National Retail foundation or Federation, nrf, whatever that is. Like, like they would gather like three metrics like conversion rate and you know, whatever from there their members. But like what was their incentive? Like, well that was so they could publish a book once a year that would have these three metrics in it and that would be part of their justification for their members to kind of re up. But it doesn't seem like there's, there's a totally objective and altruistic market out there in the business world saying we're going to go through all this work to minimize the noise in benchmarks around a handful of metrics like Qui Bono, like who, like who, who benefits from that? So that just goes back to questioning the usefulness of them.

[Mo]

Yeah, well, we're definitely not going to get through this episode without me being able to have a little bit of a cathartic moment about my most hated, least helpful benchmark, that in my previous role when I was very focused on experimentation, a client didn't go by where we didn't have to address it. So let's see if anyone can finish this sentence. Oh, even best in class experimentation programs have a win rate of.

[Eric Sandesham]

95%.

[Mo]

30. It's low. 30. 30.

[Eric Sandesham]

That's not close at all. Sorry, I thought you were going to.

[Mo]

Say oh, anyway, yeah, but that win Rate of 30% Best in Class optimization experimentation programs, that, that's like. And I don't know who said it first, but we kind of, the industry kind of like rallied around it. I'm telling you, Google it, you'll find everyone references that point. But there is no relationship between win, win rate and how much smarter you're making your organization by taking that hypothesis led mindset or using controlled experimentation to de risk decisions. And so it would just irk me to the umpteenth degree about, well, let's, let's put this down as, you know, benchmark against the 30%. Okay, let's have a more meaningful.

[Eric Sandesham]

I never heard that.

[Mo]

Really? Mo, I'm so wrong.

[Eric Sandesham]

I like the thing I always hear is like you need to have a 95% confidence interval. Like the thing I always hear.

[Mo]

Yes, you always hear that too, for sure. But the benchmark of win rate, I've.

[Eric Sandesham]

Never heard the 30% win rate, but I don't know, maybe I haven't been doing enough experimentation.

[Mo]

Not helpful.

[Eric Sandesham]

So Val, how did you handle that? How did you address that with all the clients?

[Mo]

It was all about like, do we think that that really has any relationship with the more meaningful metrics about why you're making this choice to invest in experimentation? It's the same thing as like the, the false relationship between MPS and revenue. Like there was no, there's no predictability or relationship between those two things. And so like, let's decouple those concepts and see how we can make sure we're putting smart inputs into the machine to make sure that again, we're testing what matters to the business and things that are going to help move things forward versus like, well, you know, I can test these button colors over here without getting legal approval. So let's push those 30 tests through. Right.

[Eric Sandesham]

So.

[Tim]

You know, I briefly, before we started this show, I thought, are we going to be able to talk for a whole show about benchmarks? And I, and we have mainly because.

[Eric Sandesham]

I clearly did not understand what benchmarks are. So that's been a helpful place.

[Tim]

It brought up But I think Eric nailed it. Like, it is a word that is like, oh, this is a plain word and it does get contorted by different people can mean. Which is. Is another whole area where it can. We can get in trouble if we're not talking about the same thing. I could get labeled as the person who hates benchmarks and somebody's actually thinking, I hate market research.

[Eric Sandesham]

So I've realized, like, to be honest, through the course of this conversation, I've realized when I talk to finance and they say benchmarks, they mean market research. That is like been my epiphany in this conversation. And we are often working on things together. And now I'm like, oh, I need to reframe this. So this is been very helpful, Eric.

[Tim]

Well, we are. I've. There. There are more things I would love to fetch about, but I am sitting in Michael Helbling's seat and he wants it back. So we're gonna have to start to start to wrap before we close out. We always like to do last call, go around and have everyone share a thing or two that they found interesting related to benchmarks or not. Hopefully it's an above baseline quality last call, but if not, that's okay too. So, Eric, you're our guest. Do you want to share the first last call?

[Val Kroll]

Sure, sure. Okay. But it's not, it's not related to benchmarks.

[Mo]

Okay.

[Tim]

Totally.

[Mo]

Mine's not either.

[Val Kroll]

A little bit about expected. Yeah. So this was an article. Article, you know, I, I read on Medium, which I post my articles on as well. And, and you know, it's all a rage now with, with Generative AI, you know, artificial general intelligence. Everyone's worried that, you know, we are all going to hell in a handbasket. You know, it's a terminal event, right, where the AI wakes up and all of that. And, and this, this person on. On Medium Ro. Actually, I don't know the person's name at all because they write under a handle or a pen name and their pen name is from Narrow to General AI. That's all I see in the title of the author and the title of the blog or the article is actually a very long one. It says a theory of intelligence that denies teleological purpose. And okay, so the title was so odd that when it popped up in my inbox and Medium, I said, okay, let's check it out. It's a pretty long article, a little bit philosophical, but one of the points that they were making about why we won't get to, in the near term to Artificial general intelligence really resonated with me. And when we think of, say, AI today, we think that it will be able to reason, solve, and of course the arguments will flow ways, but clearly we're making some progress. But the author here makes a very nice, succinct argument to say, look, all of AI ultimately comes down into the space called problem solving. And you can push for it, right? You can even say, well, at some point, maybe the AI will be able to reason well enough and all of that, but it is still in the space of problem solving. But the author says, actually we are not. You know, the human experience is not defined by problem solving. In fact, a big chunk of it is defined by problem finding. And that was a huge aha moment for me. It's true. I mean, we make our own problems. Like, look at this conversation on Benchmark. We didn't have a problem before, and we define it, we shape it, we argue it. And this idea of problem finding, problem defining was a huge aha moment for me.

[Mo]

I love it.

[Val Kroll]

It says, no, AI. AI isn't built to do that.

[Eric Sandesham]

Yeah, I love that.

[Tim]

Val and I are salivating because that's kind of core to the facts and feelings process, is identifying problems and then thinking through how they might be solved. So love. I like it.

[Mo]

Love.

[Tim]

Very good. Nicely done. Val, what's your last call?

[Mo]

Sure. So mine's a twofer, but both of them are relatively quick one. I just have to give a shout out to Eric. I know you mentioned in the intro, Tim, that Eric had been doing a publishing.

[Tim]

My twofer was going to be a shout out to Eric.

[Mo]

Okay, well, guess who got to go first.

[Tim]

I guess I'm just gonna have one then.

[Mo]

Well, maybe you'll call out some different pieces. But I love the way you write too, Eric. Like the. There's a whole section in there about the problem with dashboards, the problem with data visualization, the problem with data literacy. And I just, like, love the stance that you take in the way that you break it down. And it's always, like, really succinct. And so it's really a fun read. So I've enjoyed following you and so glad that you could be our guest today.

[Val Kroll]

Thank you. Thank you for that.

[Mo]

So that's one. And Tim, if you have some specific ones, I didn't go too deep, so. So you can. You can throw. Throw some up, too.

[Tim]

Okay.

[Mo]

And then the second one is an upcoming conference that you all might have heard of, Experimentation Island. So February 26 through 28 of next year, it is its inaugural year, so Kelly Wertham and Tone Wesling are bringing it to the US The. The. The best parts of conversion hotel that happened over in Europe years ago.

[Eric Sandesham]

Is it on an island?

[Mo]

It is on an island.

[Val Kroll]

What?

[Eric Sandesham]

Maybe I need to go to this.

[Mo]

It's gonna be awesome. They're doing a lot to really make sure that the experience of the attendees is gonna be great. But it's on St. Simon's island, off.

[Tim]

Of Georgia, which there's a keynote about benchmarking. Benchmarking your win rate for your experimentation program triggered.

[Mo]

Yeah, Tim and I are speakers, so I'm super excited. Or performers. They're called performers. But yeah, there's some. Some good programming. Yeah, Tim, get into it. But I'm excited. So we're just starting doing some of the. The planning. And so it was front of mind. I just wanted to drop that for our listeners so they could play.

[Eric Sandesham]

To be clear, I was thinking, like, Hawaii.

[Val Kroll]

When you're thinking is right.

[Eric Sandesham]

Yeah, I've just looked it up. I'm like, I'm gonna have to. To mention this to Ton and Kelly.

[Val Kroll]

Benchmarks. Yeah. What islands?

[Mo]

There you go.

[Tim]

Well, what's your last call that we can then denigrate to shreds?

[Mo]

Yeah.

[Eric Sandesham]

Okay, look, I always bang on about the acquired podcast, but obviously I traveled not too long ago, and I got a snippet of time to listen to a couple of things, and it's two particular episodes that just blew my mind. One is the episode on Costco, and the other one is the second on Hermes. And like, I just love how these guys, like, really get into the history of a company. Like, there was so much stuff about Costco that I didn't know. That now makes me probably an even bigger Costco lover. And likewise, I now have this obsession with wanting to buy something from Hermes, which I never had any desire to do ever. But then that's not actually my real last call because I've mentioned that podcast many, many times. I found this Instagram called To youo From Steph, and it's really about, like, growth and personal development. And you'll see some of, like, quite common, I guess, like, sentences and posts about growth and personal development. But she's such a beautiful designer. And I don't know, I'm. I'm still trying to figure out where and how I can use it. Like, you know, it's comments, like, talking about, like, the heaviness of the load or like, what today's progress feels like. It's very personal, developing, but, like, the her posts are just so beautiful that it kind of makes you revisit some of these sentiments and I'm trying to figure out like how I can adopt. I don't know. I'm not creative or artistic so I have a lot of admiration for, for her page and just how she's getting just making revisiting some of the thoughts thoughts really nice just because of. Of how beautiful they are. So yeah, that's a bit of a random one.

[Mo]

Does she post it from an island?

[Eric Sandesham]

No, but it would be better if it came from Hawaii. Like obviously.

[Mo]

I'm excited to check it out.

[Eric Sandesham]

All right. And over to you Tim. What's your last call?

[Tim]

So I promise we do not consistently like log roll the guest but the same thing one impressed with So I was also going to note that Eric, your weekly posts and they're very consumable but the one specifically because we had a listener who had submitted an idea so please listeners continue to submit ideas. We have a long list and we've been getting I swear the quality of the ideas for show ideas has gone up like markedly in the last 12 months. But somebody had actually chimed in and said what about using data like at a like small company, like small data. And like literally the next day Eric had a post that was how smaller organizations can build data analytics capabilities and sort of talking about sort of there's a little bit of turning it on that head with kind of how you approach that. So wasn't exactly what that listener was kind of asking for. But so that was I again, I'm now kind of hooked on your writing.

[Val Kroll]

Thank you.

[Tim]

But my other one is it's like an oldie that's new again and I don't think I have brought it up on here but tyler vigan.com spurious correlations the OG I saw him years ago speak at an E Metrics. I mean he's, he's a fascinating guy because he's, I mean he's like a BCG consultant in like supply chain stuff. But he has completely revamped and this was maybe 6 months ago he redid the tylervigan.com same spurious correlations. The same you go there, it shows, you know, whatever two metrics that are training together. But what he added was the, the academic paper LLM generated that supports it and I mean it is fully academic paper formatted abstract two columns totally auto generated and I mean you read them, they're like maybe three or four pages and like the level of rationale explaining why shut the front door yeah, it's.

[Mo]

It.

[Tim]

I. I mean, a lot of times you see that, you're like, oh, that's like. You think like, oh, that's cute. I'm like, it's the idea. No, I've actually read a few of these because I'm like, these are so delightful to read. And the. I. I don't know where he finds the time. I'm like, that wasn't like, oh, I'm just coming up with a little. Making a little. A little chatgpt app, like the things like formatted and somehow he's got it actually pulling rationalizations for like theories.

[Mo]

That kind of was one of your favorites. They always make me laugh.

[Tim]

Yeah. And then of course you were going to ask. I logged it a while back and now of course I cannot remember Nicolas.

[Mo]

Cage movies and drownings and like people who eat cheese and divorce or something.

[Eric Sandesham]

I mean, my sister would be very supportive of that as a non cheese eater. She's like, obviously that's the end of every marriage.

[Tim]

Yeah. So with that, so I'll do my final housekeeping. And I realized I did not take my notes because Michael can usually just rattle these off. But Eric, thank you again for coming on the show. This has been a really fun discussion and I picked up. What was it? It was bushwhacked. What?

[Mo]

Bushwhacking your way through. So good.

[Tim]

Bushwhacking your way through. Yeah. Other good stuff too, but this was great.

[Val Kroll]

So thank you. Thank you for having me. It was such a wonderful conversation.

[Tim]

Yeah, awesome listeners. We love to hear from you. So reach out to us on the Measure Slack on LinkedIn. If you want to submit a topic idea with or without a proposed guest, you can do that at AnalyticsHour IO. You can also request yourself a free sticker there. So thank you for listening. If you really are motivated and want to go onto your podcast listening platform and leave us a review or a rating, that'd be kind of swell too. No show would be complete without thanking Josh Crowhurst, our behind the scenes mostly producer who makes the audio sound normal and less incoherent than it would if we published it raw. This also was kind of the engine behind our presence on YouTube, which we now have a presence on YouTube if that's your preferred consumption. And with that, regardless of whether you are listening to podcasts at a normal speed, at an above benchmark speed, at a below benchmark speed, at 2 and a half speed. For Val and for Mo. Keep analyzing.

[Val Kroll]

Thanks for listening.

[Tim]

Let's keep the conversation going with your.

[Val Kroll]

Comments, suggestions and questions on Twitter @nalyticshour, on the web at analyticshour.IO, our LinkedIn group, and the MeasuredChat Slack group. Music for the podcast by Josh Crowhurst.

[Tim]

So smart guys wanted to fit in.

[Mo]

So they made up a term called analytics.

[Tim]

Analytics don't work.

[Mo]

Do the analytics say, go for it?

[Val Kroll]

No matter who's going for it.

[Eric Sandesham]

So if you and I were on the field, the analytics say, go for it.

[Mo]

It's the stupidest, laziest, lamest thing I've ever heard. For reasoning in competition.

[Tim]

Hi, everyone. Welcome to the analytics power out. You know.

[Eric Sandesham]

Time. Wow. Wow. That. That tiredness really gets you.

[Tim]

Word number seven.

[Mo]

Okay, that's a record.

[Tim]

One more time. Rock flag and NPS rules.