Loading summary
Transcript55 lines
- [00:02]
Kurt Nickish
KPMG makes the difference by creating value like developing strategic insights that help drive M and a success or embedding AI solutions into your business to sustain competitive advantage. KPMG make the Difference. Learn more at www.kpmg.us insights. Welcome to HBrown Strategy Case Studies and conversations with the world's top business and management experts. Hand selected to help you unlock new ways of doing business. Fueled by the promise of concrete insights, organizations are now more than ever prioritizing data in their decision making process. But it can go wrong. Many leaders don't understand that their decisions are only as good as how they interpret the data. Today, Professor Michael Luca of Johns Hopkins Carey Business School and Professor Amy Edmondson of Harvard Business School will share a framework for making better decisions. By interpreting your data more effectively, you'll learn how to tell if the data you're collecting is relevant to your goal, how to avoid some common traps of misusing data, and and how to synthesize internal and external data. This episode originally aired on HBR IdeaCast in August 2024. Here it is.
- [01:42]
Welcome to the HBR IdeaCast from Harvard Business Review. I'm Kurt Nick. You're a business owner and you're interested in reaching out to new customers. You know that data is important. I mean, that's clear, right? So you put out a survey into the field asking what kinds of products your ideal customers are looking for. You get that data back and you have a clear decision made for you as to which direction to go. You develop and sell that new product with a big marketing push behind it, and it flops. But how can the data be wrong? Today's guests believe in data, of course, but they see major ways in which over reliance or under reliance on studies and statistics steer organizations wrong. They found that leaders often go to one of two extremes, believing that the data at hand is infallible or dismissing it outright. They've developed a framework for a better way to discuss and process data, to interrogate the data at hand. Michael Luca is a professor at Johns Hopkins Carey Business School and Amy Edmondson is a professor at Harvard Business School. They wrote the HBR article Where Data Driven Decision Making Can Go Wrong. Welcome. Thanks so much to both of you.
- [03:06]
Amy Edmondson
Thanks for having us.
- [03:07]
Michael Luca
Thank you.
- [03:14]
Kurt Nickish
Are business leaders relying too heavily on data to make decisions?
- [03:19]
Amy Edmondson
I don't think that's quite the problem. One of the things that really motivated Michael and me to get together is that I study leadership and leadership conversations, particularly around really difficult, important decisions. And Michael is a data science expert and Our mutual observation is that when leadership teams and leaders are using data, or teams at any level are using data, they're often not using it well. And so we've identified predictable or frequent errors and our idea was to help people sort of anticipate those and thereby do better.
- [04:05]
Kurt Nickish
Is it more of a data science understanding problem here or more of a having the right culture to discuss the data correctly?
- [04:14]
Amy Edmondson
Well, that's just it. We think it's both. But I'll just say, in a way, my side of the problem is we need to open up the conversation so that it's more honest, more transparent. We are in fact better able to use the data we have, but that's not enough. And that's a lot. But just getting that done will not ensure high quality data driven decision making.
- [04:39]
Kurt Nickish
Mike, data has kind of been all the rage, right, for at least the last decade. I feel like it was 10 years ago or so that Harvard Business Review published this article saying that data scientist was the sexy new job of the 21st century. A lot of places make a priority of data to have something concrete and scientific. If they're getting better at collecting and analyzing data, where's the decision making problem here?
- [05:06]
Michael Luca
We're certainly surrounded by data. There's growing data collection at a wide variety of companies. There's also growing research that people are able to tap into to try to get a better sense of what the broader literature says about questions that managers are grappling with. But at the same time, it's not really about just having data. It's about understanding both the strengths of the data that you have and the limitations and being able to effectively translate that into managerial decisions. There are a couple of challenges that we discussed in the article, but they all kind of come down to this idea of once you see an analysis, and the analysis could be coming from within your company or from something that you've read in the news or from a research paper. How do you take that and understand how that maps to the problem that you have at hand? And that's the decision challenge. And this is where effective conversations around data and having a framework for what questions to be asking yourself and what questions to be discussing with your team come into play.
- [06:06]
Kurt Nickish
In your interviews with practitioners, you identified that there's kind of two big reactions to this data that's been collected, internal or external. As you just said, where do those reactions come from? Why are we seeing that?
- [06:20]
Amy Edmondson
As you said, Kurt, data is the rage.
- [06:24]
Kurt Nickish
And we have more than ever.
- [06:25]
Amy Edmondson
And we have more than ever, right? So it's sort of. You can really understand the why. Okay, great. You're telling me there's the answer. Everybody should get a pay raise and that'll make us more profitable. Okay, I'm just going to do it. Or, yeah, that's nice literature out there. But really we're different. I think we see both modes and they're easy to understand. Both are wrong. Right. But both need to be more thoughtful and, you know, probing in what applies, what doesn't apply, what does this really mean for us? And we believe there are good answers to those questions, but they won't pop out without some thoughtful conversations.
- [07:03]
Michael Luca
Analytics or any empirical analysis is rarely going to be definitive. I think the conversations need to come around. What are the outcomes that we're tracking? How does it map to the things that we care about? What is the strategy they're using to know if an effect that they're saying is causal actually is? I think those conversations often don't happen, and there's a number of reasons that they don't happen in organizations.
- [07:30]
Kurt Nickish
So you're advocating for kind of this middle path here where you really interrogate the data, understand it, understand its limitations and how much it does apply to you, how much it can be generalized, which sounds like work, but you've laid out a framework to do that. Let's start with that. Where the data comes from, internal or external? Why is that a key thing to understand?
- [07:53]
Michael Luca
When we think about external data, there's exciting opportunities to take a look at what the literature is saying on a topic. So, for example, suppose that you are managing a warehouse and trying to understand the likely effect of increasing pay for warehouse employees. You don't have to just guess what the effect is going to be. You could take a look and see other experiments or other causal analyses to try to get a sense of what people have learned in other contexts. And then you, as a decision maker, you could think about, how does that port over to your setting? Now, in thinking about how to port over to your setting, there are a couple of big buckets of challenges that you'll want to think about. You'll want to think about the internal validity of the analysis that you're looking at. So, meaning, was the analysis correct in the context in which it's being studied? So is the causal claim of wages on, say, productivity? Is that well identified? Are they outcomes that are relevant there? And then you want to think about the external validity or the generalizability from that setting to the setting that you're interested in, and think about how Closely those map together. So I think it's both an opportunity to look more broadly at what the literature is saying elsewhere and to bring it over to your setting, but also a challenge in thinking about what's being measured and how to port it over. Now, for larger companies especially, there's been a growth of internal data. So you could think about Google or Amazon or other large tech companies that are tracking exorbitant amounts of data and often running experiments and causal analyses. Those come with their own challenges, thinking about what is the metric we care about. So it's slightly different challenges, but related, but then sort of zooming out. What you want to think about is combining what internal and external data do we have and how do we put it all together to come to the best decision that we can, to get.
- [09:49]
Amy Edmondson
A fuller picture, really. You know, in a way, what we're saying, which is pretty simple, but I think really profound, is that you can't just assume, you know, if someone tells you, here's a result, you can't just take it at face value. You have to interrogate it. You have to ask questions about causality, was it an experiment or not? You have to ask questions about, you know, what was actually measured and what's the context like and how is it different from my context and all the rest. And these are things that. That scientists would naturally do and managers also can do and get better decisions as a result.
- [10:27]
Kurt Nickish
It's a lot of basic statistics skills, right, that not everybody has. It sounds like you kind of want that capability across the team or across the decision makers here, and not only housed in a data analytics team in your group, for instance.
- [10:45]
Amy Edmondson
Yes, and it's not that everybody needs to be a data scientist. It's that data scientists and operating managers need to talk to each other in an informed and thoughtful way. So the managers need to be able to learn and benefit from what the data scientists understand how to do. And the data scientists need to think in a way that is really about supporting the company's operations and the company's managers.
- [11:18]
Michael Luca
So maybe just one quick example, there's this famous ebay experiment that looks at the impact of advertising on Google. And what they found is largely the ads that they had been running were not effective at generating new business. Coming into ebay, and just to spell.
- [11:33]
Kurt Nickish
Out this ebay experiment, they had been advertising in markets and seeing more sales there, and they thought the advertising was working, but they were basically just advertising to people who were going to be buying more from them anyway. So the effect of all that Advertising spending was pretty muted.
- [11:49]
Michael Luca
That's exactly right. So they had been running billions of dollars of ads per year on search engine ads. So they had actually brought in consultants to look at this and try to analyze what the impact was. And initially they thought that there was a positive effect because of the correlation. But then by thinking more carefully about the fact that ads are highly targeted, that led them to run an experiment to get at the causal effect of ads. And, and that's when they realized that many of the ads they were running were largely ineffective.
- [12:18]
Kurt Nickish
And so was this a correlation causation problem? Essentially at its core?
- [12:22]
Michael Luca
So for ebay, there was a correlation versus causation problem. Then you could think about generalizing that to other settings, other types of ads on ebay, other companies that want to use this result. In fact, even within that one experiment, when you dive a little bit deeper, they found certain types of ads were slightly more effective than others. So you could find corners of the world where you think there's more likely to be an effective advertising and change your advertising strategy. So it's correlation, causation, and then trying to learn more about mechanisms or where ads might work so that you could update your strategy. Then as external companies saying, here's this new evidence that's out there. How do I take this and adjust either my advertising strategy or my approach to measuring the impact of advertising?
- [13:13]
Kurt Nickish
Tell me more about the disconnect between what is measured and what matters. We all know that you get what you measure. We've all heard that. Where do managers often go wrong here?
- [13:23]
Michael Luca
Such a challenging problem. And actually earlier we were discussing the fact that many things are measured now, but many more things are not measured. So it's actually really hard to think about the relationship between one empirical result and the actual outcomes that a company might care about at the tail end. So, for example, so imagine you wanted to run an experiment on a platform and change the design. You change the design and you see more people come. That's one piece of the puzzle. But you really want to see what's the long run effect of that. How many of the customers are going to stick with you over time? How happy are they with the products or the engagement on the platform? Are there going to be other unintended consequences? And those are all really hard things to measure. We're left in a world where often analyses are focused on a combination of important things, but also things that are relatively easy to measure, which could create pretty important disconnects between the things that are measured in an experiment or an Analysis and the outcome of interest to a manager or an executive.
- [14:28]
Kurt Nickish
Amy, when you hear these problems like disconnects, you could also call that, you know, miscommunication.
- [14:35]
Amy Edmondson
Absolutely.
- [14:36]
Kurt Nickish
From an organizational culture perspective, how are you hearing this?
- [14:39]
Amy Edmondson
So I hear it as a. I think there's a general need to go slow, to go fast, and there's a strong desire to go fast. You know, just in everything, you know, data. It's a modern world. Things are moving fast. We want to get the data and then make the decision. And we write about the fact that it's this issue we're talking about right now, that making sure that the outcome we're studying, the outcome we're getting data on, is in fact a good proxy for the goal that we have. And just that, getting that right, then you can go fast or go faster. But really pausing to unpack assumptions that we might be making, what else might this design change encourage or discourage? You know, what might we be missing? Asking those kinds of good questions in a room full of thoughtful people will more often than not allow you to surface underlying assumptions or things that were missing. And, you know, when a culture allows, when an organization's culture or climate allows that kind of thoughtful wrestling with very ambiguous, challenging uncertainty content, you'll be better off. You'll design better experiments, you'll draw better inferences from the data or studies that you do have access to.
- [16:05]
Kurt Nickish
We've talked about the disconnect between what's measured and what matters, conflating correlation and causation. Let's talk about some of the other common pitfalls that you came across in your research. One is just misjudging the potential magnitude of effects. What does that mean? What did you see?
- [16:23]
Amy Edmondson
Well, we talk about our general lack of appreciation of the importance of sample size. Certainly any statistician knows this well. But intuitively, we make these errors where we might overweight an effect that happens in a very small sample and realize that that might not be representative to a much larger. So how precise can we be about the effect that we're seeing is very much dependent on the size of the sample.
- [16:57]
Kurt Nickish
You suggest a question to ask. There is what's the average effect of the change? To get a better sense of what the real effect is, I think for.
- [17:06]
Michael Luca
Managers, it's thinking about both what the average effect that was estimated and also what the confidence interval is to get a sense of where the true effect may lie. And thinking about confidence intervals is important both before and after you conduct an analysis. Before you conduct an analysis, anticipating the uncertainty in effects is going to tell you how large of a sample you might need. If you're going to, say, run an experiment after an analysis, it could tell you a little bit about what the range of true effects may be. So a recent paper looked at advertising experiments for a variety of companies and found that many of the experiments that were being run didn't have the statistical power to determine whether it had positive or negative ROI.
- [17:59]
Amy Edmondson
So they'll hear, okay, it was up 5. Sales were up 5%. Oh, great, let's do it, let's roll it out. But in fact, that up 5% was well within what's called the margin of error and may in fact even be negative. It's possible that that advertising campaign reduced interest in buying. Right. We just really don't know. Based on the sample size.
- [18:24]
Kurt Nickish
Overweighting a specific result is also a common trap. Can you explain that?
- [18:31]
Amy Edmondson
Yeah, it's a confirmation bias. Yeah. Or a desirability effect. Or sometimes if a result is just very salient or it kind of makes sense, it's easy to just say, okay, this is true, without pressure testing it, asking, what other analyses are there? What other data might we need to have more confidence in this result. So it's kind of a variation on the theme of the magnitude of the effect.
- [19:03]
Kurt Nickish
One common pitfall is also misjudging generalizability. Why is this problematic?
- [19:09]
Michael Luca
So we talk about that example in the article where there's an SVP of engineering that was talking about why he doesn't use grades in hiring and says, well, Google proved that grades don't matter. Now let's put aside the fact that we don't know how Google exactly did this analysis and whether they actually prove that it doesn't matter in the Google context. But it's a pretty big leap to then say, because they've shown this in one context, that that's going to be poured over exactly. To the context that the SVP was thinking about in his company. So I think what we have in mind here is just thinking a little bit more about the relevance of findings from one setting to the other, rather than just kind of porting it over. Exactly. Or dismissing it altogether.
- [20:01]
Kurt Nickish
What's a good strategy to break out of that when you're in that situation or when you see it happening.
- [20:06]
Amy Edmondson
Yeah. So you can't see me smiling, but I'm smiling ear to ear. Because this really falls squarely in my territory because it's so related to, you know, if you want something to be true, it can then be even harder to tell the boss. Well, hold on here. We don't really have enough confidence. So this is really about opening the door to having high quality conversations about what do we know? Really? Curiosity led conversations. What do we know? What does that tell us? What are we missing? What other tests might we run? And if X or if Y, how might that change our interpretation of what's going on? So this is where we just, we want to help people be thoughtful and analytical. But as a team sport, we want managers to think analytically, but we don't want them to become data scientists. We want them to have better conversations with each other and with their data scientists.
- [21:08]
Kurt Nickish
In teams, as data is being discussed, how, as a leader, can you communicate the importance of that culture that you're striving for here? And also how, as a manager or as a team member, how can you participate in this and what do you need to be thinking about as you talk through this stuff? Because it's definitely a process, right?
- [21:29]
Amy Edmondson
Right. I mean, in a way, it starts with framing the situation or the conversation as a learning problem solving opportunity. And I know that's obvious, but I have found if that's not made explicit, especially if there's a hierarchical relationship in the room, people just tend to code the situation as one where they're supposed to have answers or they're supposed to be. Right. And so just really taking the time, which can be 10 seconds to specify that, wow, this is a really uncertain and fairly high stakes issue for our company and it's going to be important for us to have the best possible bet we can. So what do we know and what are the data telling us and what do we need to learn? And really probing the various people in the room for their perspectives and their interpretation. So I think starting with that stage setting and then like we write about leaning into questions, you know, we provide a set of sample questions. And they aren't the only questions or even a cookbook of questions, but they illustrate the kinds of questions that need to be asked. Tone matters. Tone needs to have a feeling of genuine curiosity, like, ooh, what outcomes were measured? Not well, what outcomes were measured? You know, were they broad enough? No, it's, you know, how broad were they? Did they capture, you know, any chance that there were some unintended consequences and so forth. So it's got to be approached in a spirit of genuine learning and problem solving and viewing that as a team sport.
- [23:11]
Kurt Nickish
When can you lean into the answers?
- [23:16]
Amy Edmondson
Right? Because there's never gonna be the sort of perfect answer. The crystal ball. There are no crystal balls. So it's a very good question.
- [23:26]
Kurt Nickish
It seems like to be really good at data driven decision making, you have to be analytical and you have to have those hard skills. You also have to have the soft skills to be able to lead these discussions among your team and do it in a psychologically safe space. I mean, it definitely sounds hard.
- [23:45]
Amy Edmondson
Yes.
- [23:46]
Kurt Nickish
And you can see why a lot of people go the easy route and say, oh, that doesn't apply to us, or yes, that's the gospel truth. What's your hope out of all of this?
- [23:58]
Amy Edmondson
Well, I think my hope is that we all get more comfortable with uncertainty, start to develop the emotional and cognitive muscles of learning over knowing. Right. Embracing learning over knowing and then using the team. This is a team sport. So that. Right. That's kind of. Those are mindset things. And then so that we get more comfortable with a mode of operating that is really just test and iterate. You know, what do we try? What data? What did the data tell us? You know, what should we try next? Like life and work in kind of smaller batches rather than these giant decisions and giant rollouts. There's going to be more navigating the uncertainty, I think, going forward. And we need people who are, as you said, analytical, but also curious, also good at listening, also good at leading a team conversation so that you actually can get somewhere and it doesn't have to take forever. We can have a conversation that's quite efficient and quite thoughtful and we get to a sufficient level of confidence that we feel now we're able to act on something.
- [25:08]
Michael Luca
People talk a lot about things like big data or large scale analytics, and I think there are a lot of interesting innovations happening there. But I also think there are lots of contexts where a little bit of careful data could go a long way. So I think when it comes to many managerial questions, thinking about is this a causal inference question and if so, what is the question we're trying to answer from a team perspective? My hope is that people will be focused on trying to answer a question that could then inform a decision. And by thinking about the analytics underlying it and being comfortable with uncertainty, you get to a more effective use of data. And that's both the internal data that's sitting within your organization, but also the growing amount of external data that's coming from academic research or news articles and thinking about how to synthesize information from these different sources and then have good group discussions about how to effectively use it.
- [26:06]
Kurt Nickish
Mike and Amy, this has been great. Thanks so much for coming on the show to talk about your research.
- [26:11]
Amy Edmondson
Thank you.
- [26:12]
Michael Luca
Thanks.
- [26:18]
Kurt Nickish
You just heard Michael Luca of Johns Hopkins Carey Business School and Amy Edmondson of Harvard Business School in conversation with Kurt nickish on HBR IdeaCast. We'll be back next Wednesday with another handpicked conversation about business strategy from the Harvard Business Review. If you found this episode helpful, share it with your friends and colleagues and follow our show on Apple Podcasts, Spotify, or wherever you get your podcasts. While you're there, be sure to leave us a review. And when you're ready for more podcasts, articles, case studies, books and videos with the world's top business, business and management experts, find it all@hbr.org this episode was produced by Mary Dew and me, Hannah Bates. Ian Fox is our editor. And special thanks to Maureen Hoke, Erica Truxler, Ramsay Kabaz, Nicole Smith, Anne Bartholomew and you, our listener. See you next.