Dr. Joel Shapiro | Northwestern | The Grocery Prediction Case - It's Not Just About the Data

Summary

CMO Confidential Podcast - Detailed Summary

Episode: Dr. Joel Shapiro | Northwestern | The Grocery Prediction Case - It's Not Just About the Data
Release Date: May 20, 2025
Host: Mike Linton
Guest: Dr. Joel Shapiro, Professor at the Kellogg School of Business, Northwestern University

---

Introduction

In this episode of CMO Confidential, host Mike Linton engages in an insightful conversation with Dr. Joel Shapiro, a seasoned expert in decision science and data analytics. Dr. Shapiro, with a robust background encompassing public policy, business analytics, and law, delves into the intricacies of leveraging data to solve complex business problems. The focal point of their discussion is the Euro Grocer Case, a compelling study that underscores the challenges and nuances of data-driven decision-making in the retail sector.

---

Understanding Data and Decision Sciences

Grounding the Conversation
Dr. Shapiro begins by elucidating his journey from public policy to business, emphasizing the dynamic nature of business environments where data-driven decisions can rapidly impact outcomes. He articulates the distinction between large-scale "home run" ideas and the incremental, data-informed decisions that aggregate to significant improvements over time.

"I love the fact when data helps us make little decisions better on a daily, hourly, weekly basis because those things can aggregate pretty quickly and I like that pace."
[03:01] Dr. Joel Shapiro

Marketing vs. Other Business Functions
The discussion transitions to the comparative maturity of data utilization across business functions. Dr. Shapiro highlights marketing and operations as the most advanced in data-driven practices, contrasting them with sales, which he observes as less receptive to data interventions.

"Marketing really mature. Sales, not so much."
[05:08] Dr. Joel Shapiro

---

The Euro Grocer Case: A Deep Dive

Case Overview
Euro Grocer, a pseudonym for a major European grocery chain, faced an $8 billion annual sales landscape alongside a $250 million problem: inaccurate inventory ordering. This discrepancy led to both overstocking, resulting in spoilage, and understocking, causing lost sales.

Data Science Intervention
To address this, Euro Grocer enlisted a top-tier data science consultancy to predict category-wise demand across their stores. A pilot test revealed that the predictive model could potentially save $106 million annually by optimizing inventory levels.

"If I told you that you could get six and a half dollars back for every dollar that you spend, that's going to get your attention."
[11:42] Dr. Joel Shapiro

Board Resistance and Underlying Issues
Despite the promising pilot results, Euro Grocer's board declined the $16 million annual investment required to scale the solution. Dr. Shapiro identifies three critical factors contributing to this refusal:

1. Organizational Misalignment:

   • The predictive model suggested precise inventory allocations, but Euro Grocer's existing supplier contracts and logistics systems lacked the flexibility to implement these recommendations effectively.

   "So you can tell a grocery store that I can predict exactly how many avocados people are going to want to buy next week... you can't take full advantage of that, you know, predictive accuracy."
   [13:14] Dr. Joel Shapiro

2. Model Transparency:

   • The consultancy employed a neural network model, prized for its accuracy but notorious for its lack of interpretability. Decision-makers struggled to understand the rationale behind specific inventory recommendations, leading to distrust.

   "When somebody in a decision-making position says to you, explain to me why we should do this, when your only response is, because the model tells me to. Oof."
   [15:57] Dr. Joel Shapiro

3. Data Trust Issues:

   • Errors in the initial data set, such as incorrect records of sales promotions, eroded trust among store managers. Even minimal data inaccuracies can significantly impact stakeholders' confidence in the data-driven solutions.

   "Once people lose faith or trust in the data, it's really hard to get it back."
   [16:26] Dr. Joel Shapiro

These factors collectively underscored the complex interplay between data accuracy, model transparency, and organizational readiness, ultimately preventing the adoption of a seemingly lucrative data solution.

---

Asymmetrical Risk in Data-Driven Decisions

Understanding Asymmetrical Risk
Dr. Shapiro expands the conversation to the concept of asymmetrical risk, using a case from the Illinois Department of Child and Family Services. A predictive model intended to identify at-risk children failed catastrophically by not flagging two children who tragically lost their lives, leading to the abandonment of the tool despite its broad effectiveness.

"There is a false positive and a false negative, but the consequences of one bring one terrible outcome to the kids."
[27:09] Dr. Joel Shapiro

Lessons on Managing Asymmetrical Risks:

1. Cost of Errors:

   • Business leaders must thoroughly understand and weigh the consequences of false positives and false negatives to make informed decisions that align with their risk tolerance.

   "Decision makers need to understand deeply the cost of being wrong, the benefits of being right."
   [28:36] Dr. Joel Shapiro

2. Balancing Model Precision and Organizational Capacities:

   • Predictive accuracy must be matched with the organization's ability to act on insights, ensuring that data-driven recommendations are feasible and sustainable within existing operational frameworks.

---

Best Practices for Data-Driven Leadership

Building Trust Through Transparency and Communication
Dr. Shapiro emphasizes the pivotal role of trust in successful data initiatives. Establishing clear expectations and maintaining transparency about data limitations are crucial for fostering stakeholder confidence.

"If you're going to be successful with data, you got to set the right expectations and you got to be transparent with things when they are imperfect."
[22:46] Dr. Joel Shapiro

Effective Communication Strategies:

1. Know Your Audience:

   • Tailor communications to address the specific concerns and interests of different stakeholders, ensuring that data insights are relevant and actionable.

   "Know exactly what the purpose of your communication is... know what your audience cares about."
   [22:51] Dr. Joel Shapiro

2. Purpose-Driven Messaging:

   • Clearly define the objectives of your data presentations, whether it's to persuade, inform, or garner support, to enhance the impact of your message.

Cultivating Data Leadership:

• Dr. Shapiro advocates for the development of data leadership skills, urging data teams to collaborate closely with business units and focus on solving problems rather than dictating solutions.

  "Practical advice is for the business teams maybe be a little bit more tolerant to the data teams... data teams to be really good at thinking more deliberately about how to influence business people."
  [34:43] Dr. Joel Shapiro

---

The Role of AI and Future Implications

AI and Transparency Challenges
The conversation touches on the burgeoning role of AI in decision-making, highlighting the ongoing struggle between model complexity and the need for transparency. Dr. Shapiro expresses concern over the potential for over-reliance on large language models (LLMs) without adequate human oversight.

"AI has some adult supervision, you would feel comfortable."
[19:53] Dr. Joel Shapiro

Ensuring Responsible AI Use:

• Emphasizing the importance of human accountability and deliberate oversight to prevent misuse of AI-generated insights, particularly in high-stakes environments.

---

Final Takeaways

Key Insights for Business Leaders:

1. Data Doesn't Make Decisions:

   • Ultimately, human decision-makers are responsible for interpreting and acting on data insights. Ensuring that data is meaningful and actionable is paramount.

   "Data doesn't make decisions, People make decisions. And data needs to be helpful to them."
   [31:57] Dr. Joel Shapiro

2. Trust and Transparency:

   • Building and maintaining trust through transparent communication and setting realistic expectations about data capabilities and limitations.

3. Aligning Data Models with Organizational Capabilities:

   • Ensuring that predictive models are compatible with the organization's operational flexibility and strategic objectives.

Practical Advice:

• Foster a collaborative culture where data teams and business units work synergistically.
• Invest in data leadership training to bridge communication gaps and enhance the influence of data-driven insights.
• Always consider the human element in data interventions, recognizing that trust and understanding are as critical as technical accuracy.

---

Closing Remarks

Mike Linton concludes the episode by reiterating the essential lessons from the Euro Grocer Case and Dr. Shapiro's expertise. Listeners are encouraged to integrate these insights into their own data-driven strategies, ensuring that robust analytics are complemented by effective leadership and communication.

"Whatever else we discuss on other shows, like artificial intelligence... what's important is to remember the human element in data-driven decisions."
[35:10] Mike Linton

---

Stay Connected:
For more episodes and insights, subscribe to CMO Confidential on Spotify, Apple Podcasts, YouTube, and the I Hear Everything network.

---

Transcript

[Dr. Joel Shapiro]

The CMO Confidential Podcast is a proud member of the I Hear Everything Podcast Network. Looking to launch or scale your podcast, I Hear Everything delivers podcast production, growth and monetization solutions that transform your words into profit. Ready to give your brand a voice then visit iheareverything.com welcome to CMO Confidential, the podcast that takes you inside the drama, decisions and choices that go with being the Head of marketing. Hosted by five time CMO Mike Linton.

[Mike Linton]

Welcome marketers, advertisers and those who love them. The Chief Marketing Officer, Confidential CMO Confidential is a program that takes you inside the drama, the decisions and the politics that go with being the head of marketing at any company and what is one of the most scrutinized jobs in the executive suite. I'm Mike Linton, the former Chief Marketing Officer of Best Buy, ebay, Farmers Insurance and Ancestry.com here today with my guest, Dr. Joel Shapiro. Today's topic, the Euro Grocer Case. It's not just about the data Now. Joel is a professor at the Kellogg School of Business at Northwestern where he has taught decision Science for nearly 11 years after stint as an Associate Dean. Prior to that he was a doctoral fellow at Rand Corporation and to top it off, he also has a degree in law. Welcome Joel. We could talk to you about almost anything.

[Dr. Joel Shapiro]

Yeah, you could talk to me about pretty much anything. I don't know if I didn't have any good answers though. And the data stuff, I'm pretty good, so that's good.

[Mike Linton]

Let's go into the data stuff before we dig into this case. Let's talk about ground our listeners in what you teach and why. Like tell us about economics and decision sciences.

[Dr. Joel Shapiro]

Yeah, sure. So I'm happy to. So my background is really as like a data guy in public policy, so I did my PhD in policy analysis because I was always really interested in social systems and social science and how we can use data to make people's lives better. And that's kind of what I do, except I made a bit of a transition about 12 years ago into business. And you know, for me, business moves fast, businesses are accountable. When I was doing work in policy, it was too easy to write a report and it sits on a shelf and you know, policy and government kind of moves slowly. So I found that a little frustrating. So really what I do now is I teach students how to use data and data science and analytics to make better decisions in business. And we move fast and make quick decisions and test things and hopefully they're innovating along the way, which Is, you.

[Mike Linton]

Know, I was going to ask you about why you didn't go to politics or medicine or something and you just answered that, which is I want to be, I want to be teaching people about decisions at that make things better at speed. Is that fair?

[Dr. Joel Shapiro]

Yeah, I mean, I think sometimes, you know, we get a little bit too enamored with, you know, the big home run ideas, which is fine. Like big home run ideas are good. The issue that I have is sometimes data helps us make little better decisions and over time those aggregate pretty quickly. The problem with home run ideas is sometimes they work and sometimes they don't and they can take a long time. But I love the fact when data helps us make little decisions better on a daily, hourly, weekly basis because those things, as I said, they can aggregate pretty quickly and I like that pace. And when you actually measure things, you can see that improvement over time, which helps give credence to this whole thing.

[Mike Linton]

I think this, well, one, it's a lot like a 401k. You put in a little bit of money every year and suddenly there it is. And I think this is very apropos to marketing. And so, you know, when you look at the entire business landscape and you look at this use of data and decision sciences, talk to us about marketing versus other functions, just in general writ large about using data to drive decisions.

[Dr. Joel Shapiro]

So I think of marketing as one of two functions in a business that is pretty mature around the use of data. I think operations is another one. So, you know, when people are in operations, they tend to think, my experience, they tend to think like an engineer. They know how to build strong systems that sort of everything works really well together. You can get some reliable output out of it. Marketing to me is really good at using data for social systems. Systems that are comprised of people who do measurable things and they're measurable outcomes and the use of data to sort of inform what really works. Pretty mature. You know, I was the, I was the chief analytics officer at a sales software company and when I started there, I made this terrible assumption and I was like, well, I bet salespeople use data really well because marketers use it really well. And in my mind I was sort of connecting them. That was like the worst assumption I could have made. Marketing really mature. Sales, not so much. And other parts of the business sort of vary as well.

[Mike Linton]

And it's part of that. Before we get to the ur Grosser case, it's part of that because the way the data structured and absorbed by the salesforce and the Marketer. Because if I'm sitting there, I have all this CRM data, all my search data, you know, my digital data, and the salesforce sometimes doesn't have that data. Is, is that the reason or is is there another reason why sales. Why you made the wrong assumption on sales?

[Dr. Joel Shapiro]

I think it has more to do with the fact that people in sales pride themselves on knowing how to work with people, and they've simply been more resistant to input from, you know, some sort of artless algorithm, you know, something like that. I think it's more that people are a little bit more defiant in sales and say, no machine is going to tell me how to sell. I know how to sell. I've been doing this for a long time. And for whatever reason, you know, marketing measured early. They came up with really good ways of measuring success and attribution and running experiments. And I wish I had. I wish I had cracked the nut on why sales hasn't done as much. But I think it's just sort of this. I know how to sell. Don't let the data tell me. Don't tell me how to do it differently.

[Mike Linton]

Talk to me about those heartless, nefarious algorithms. Even those are for the marketers. Keep them in the back room. All right, so let's sketch out the case for the listeners, the Euro Grocer case. And Euro Grocer is a pseudonym for a real company. Even if you're Sherlock Holmes, you probably can't take a guess. But, Joel, let's start with the business problem and then the data science solution.

[Dr. Joel Shapiro]

Yeah, yeah. So I'm happy to. So Euro Grocer, as you said, it's a pseudonym for a real company. Not surprisingly, it's a European grocery store chain and super interesting case. I teach with it a lot and it sounds sort of mundane on its face, I think, but there's so many layers to it that teach us some really interesting things. So here's the summary of the case. Euro grocer does about $8 billion a year in sales, and they had about a $250 million problem. And that problem was that they were ordering the wrong amount of stuff for their shelves. Sometimes they would have too much and it would spoil, and sometimes they wouldn't have enough. And both of those are problematic. Right? You don't have enough, you forego profit. You have too much. You bought things that you didn't need to buy.

[Mike Linton]

And we also know that the grocery margin is, you know, I mean, I recall when I was in retail, you know, if you can make over a 2. You were killing it. Yeah, like, and that's 2%. That stats of 2%. Not. So you're killing it. So 250 million on 8 billion, that is a lot, a lot of money. Tell me if I'm getting that wrong, but that's my recollection of grocery.

[Dr. Joel Shapiro]

No, that's exactly right. And it's so funny you say that because I get some people when I teach this case and I'm like a $250 million problem. And they'll be like, yeah, but they're 8 billion. It's only like 3%. So why did they even care? I'm like, well, where I come from, $250 million is a pretty big number. So they care. Right. So that's the problem. They're not ordering the right amount of stuff. Okay. And this is a great data problem because if you could predict demand, you could arguably order the right amount of stuff. Yeah, right. And so what they do is they hire a data science consultancy to come in, a consultant firm to come in and hey, can we actually predict demand? So that's sort of the context here. They bring in this consultant to try and predict demand and they have a really high quality data science consultancy. And the very first thing that they do is they run up a little pilot test. Not everything is predictable. Like, don't spend a lot of money and resources hiring somebody to do this until you test it out first. Because it's not predictable. You don't want to be, you know, all in with a big price tag. Turns out it's pretty predictable. Demand is pretty predictable. Not perfectly so. Nobody's ever going to make the claim that you can know exactly how much to have on which shelves and blah, blah, blah.

[Mike Linton]

But it's also, it's also just if I want to make sure everyone's getting this right because I would say it's grocery. So the market isn't like wildly expanding and contracting for something. It's kind of, you know, you got your population around your stores, you're delivering all this stuff. You probably have an awful lot of your own first party data right here. So me is a, is a great data feed for, for somebody.

[Dr. Joel Shapiro]

Yeah, exactly. So what they do is they try and predict the demand of different product categories at each one of these stores. How much, you know, stuff are people going to buy in bakery at this one store and dairy over here and meat and seafood and so forth. And that's sort of the level at which they do the prediction. Can we predict category demand? And they have like 25 categories per store. Can we predict category demand at each one of these stores? So they do a little pilot test and it turns out, as I said, they couldn't do perfect prediction. There's no way they're going to solve this whole $50 million. But the pilot test shows that they can get 42% of the way there.

[Mike Linton]

That's, that's, that's real money. When you're, you're working, I mean, you're, you're sitting over a 1% profit gain.

[Dr. Joel Shapiro]

Yeah, yeah. It's $106 million improvement that, that this pilot test suggests. If you roll out this prediction more broadly, looking at $106 million. Okay. Next thing that you want to do as a business leader, if somebody says to you you can do Something that has $106 million benefit, I'm pretty sure your next question is how much is it going to cost me?

[Mike Linton]

Yes. And, but then generally, how much is it going to cost me? And under the assumption that we're not going to have to build something super new, how fast could we do it?

[Dr. Joel Shapiro]

Yeah, yeah. So really good question too. Um, capabilities were pretty much right there. They felt good about building it fast and the price tag that they put on it was about $16 million. And I just want to, I want to pause on that for a second. So $106 million benefit for a $16 million investment. And this is annual. So 106 million is an annuity.

[Mike Linton]

And the 16 million is a one time cost.

[Dr. Joel Shapiro]

$16 million is going to be an annuity too.

[Mike Linton]

All right, so you're sitting there at a 64.

[Dr. Joel Shapiro]

Yeah.

[Mike Linton]

You're sitting there at a pretty big, pretty big gain.

[Dr. Joel Shapiro]

Yeah. I mean, there's a couple million dollars of upfront costs, but if you look at it yearly, it's about $16 million. Now do the math on that. You can do the math. It's a six and a half times return.

[Mike Linton]

Yeah. So it pays off in like two months.

[Dr. Joel Shapiro]

I mean, it's the biggest no brainer ever. Now keep in mind this was a pilot test and we're talking about extrapolating these results to the broader enterprise. But if I told you that you could get six and a half dollars back for every dollar that you spend, that's going to get your attention, you know, that's going to get your attention. And what's interesting about this story is that, you know, we could stop right here. We could be like data, it's amazing, and analytics, and we could all raise our hands in victory and be Excited about it, but that's not the way the story goes. So the Euro grocer board looks at this and they consider this $16 million a year investment and they say, thanks, but no thanks. We're going to pass. Come on, what you're going to pass on. Why would you possibly pass on an investment like that? And the reasons why they passed on it are where this turns fascinating and where there are a lot of lessons to be learned. And I can share those with you, unless you want to.

[Mike Linton]

Oh, no, we want, we want to hear. Because this is where the, the data science consulting company is tearing its hair out and saying, this is one. I can't believe you're not going to do this to. This was a case study we're going to share from stages all over the world about how great we are at consulting. Yeah, we're not going to do it. So tell our list why it didn't get done.

[Dr. Joel Shapiro]

Yeah. So there are really three big things here that I'll highlight for you. So one of them is a little bit of the misalignment with the results of the data model with the realities of the business. And what I mean by that is you can tell a grocery store that I can predict exactly how many avocados people are going to want to buy next week and the week after. And you should make sure to order different amounts. But if you locked into some supplier contracts or if you've got some logistics processes that don't allow you to have that flexibility, you can't take full advantage of that, you know, predictive accuracy. I sometimes refer to this as being a model driven organization. You want to have your organization set up so that you can use the results of a quantitative model that's really important to being successful with data. And in this one way, Euro grocer couldn't really adapt to the accuracy and the flexibility and the sort of periodic changes in those predictions. So that was one, that was one big issue. You with me?

[Mike Linton]

Yeah. I mean, that says I know the answer, but I just can't do it. Yeah, I need to get 100 avocados for store A and 700 for store B. But since I can't actually get that done, I'm going to get 400 and send 400 to each store. Yeah, I'm going to get 800 and send 400 each store. Yeah, yeah.

[Dr. Joel Shapiro]

And, and you know there's a cost to that, right? It's nothing more than just additional cost. But now the, the board is sort of like, huh, man, we can't really do that. So maybe this isn't quite as good as it looks. The second thing that they have a problem with is they character, the data science team. And they're like, okay, you're telling us that we need this many avocados here and this many here. Why is that? So that's a challenge for people who work with data models. When we build out data models, oftentimes, not always, but oftentimes there's a trade off between accuracy of models and transparency of models. So what they did here, the data science consultancy here, built what's called a neural network. You might have heard of it. It's a really complex, complicated, technically advanced way of building a predictive model. And it's really good in accuracy and it's really bad in transparency. And when somebody who's in a decision making, a position of decision making authority says to you, explain to me why we should do this, when your only response is, because the model tells me to. Oof. That's a challenge.

[Mike Linton]

That is because you'd be saying, look, why is this one store sticking with the avocado strategy going, or example, why are they selling seven times more avocados than this other store that's only 70 miles away?

[Dr. Joel Shapiro]

Yeah.

[Mike Linton]

And you're 70 kilometers away since we're doing a European grocery. So. Yeah, yeah.

[Dr. Joel Shapiro]

So. So, you know when people ask you why and you can't answer why, sometimes that can be that, sometimes that can be a problem. And this is always an issue with people in data. Are we building models to be maximally useful or transparent? And when those two things are intentioned, one of them has to give and it's not always clear which one of them it should be.

[Mike Linton]

Okay, you said there was a third thing, I think. Did we get them all out?

[Dr. Joel Shapiro]

No, we didn't. So the third is so fascinating and it just speaks to such a broader sort of area and concern that I have with data. When the store managers were brought in on this, they saw that parts of the data didn't quite look right to them. It was like, oh, this shows that orange juice was on sale at this year. I've been a store manager here for 20 years and we've never put orange juice on sale. I don't think your data is right. It turns out that there were some errors in the data. So the data science consultancy team goes back to it and they're like, oh, we'll fix the errors and so forth. And they fixed the errors and they reran the analysis and it all looked about the same the errors were very minimal. But once people lose faith or trust in the data, it's really hard to get it back. And now all of a sudden you're in a position of you told me the results the first time and the data was wrong. How do I know whether I can believe you now? And that notion of people not trusting the data I find fascinating and important and one of the sort of least understood or remedied things that we can talk about in the data space. And those three things together not being able to use the data well, the lack of transparency in the model and lack of trust in the data ended up killing it. And the board at euro grocer said $16 million is still a lot of money and we're going to do other things instead Now.

[Mike Linton]

Oh, hey, I have to do a couple follow ons on this one. The first is, having been in a lot of companies, sometimes the culture, it just wants to resist this. And if it's resisting, it will find any reason like the data or the avocado difference and it will just keep pecking away, pecking away at this until it finds a reason to not accept it. And there's other companies that are more on the forefront of oh my God, we know we got to do something that will say they'll see the hundred and over $100 million savings and they'll go, all right, there might be some errors in this, but we got to go get it. So let's overcome the errors. How much of this is culture and how much of this is the pure presentation of the. I know you can't actually split this out, but the pure presentation of the model by the data people.

[Dr. Joel Shapiro]

Yeah, it's a really good question. As you know, I don't really know the answer to this. I mean, I will say that any organization that's serious enough to bring in a top flight data science consulting firm to do this, I think at some point the culture's gotta be there. I mean, it's possible that someone, you know, K's it and then they're not involved in the decision making. So all apart, but I think those are the two issues and I don't know which one is which in this case.

[Mike Linton]

Okay, that's it. We're going to talk about what people should do in a when they're in this kind of position, how to make sure it gets through the company. But when you look at AI now and the answers AI is producing and then you want to want the transparency of AI, which to me is looks like it's really challenging to figure out any transparency. How does this speak to companies taking all the AI information they get and then absorbing it? Because it sounds like this grocer would say, well, I can't possibly see anything in this giant LLM model. I'm not going to do anything. What do you think is going to happen on that front?

[Dr. Joel Shapiro]

Well, look, I mean, I think one of the nice things about LLMs is that if you review what an LLM tells you, a large language model tells you that you should do something. I think we're still at the point where a lot of people don't sort of just blindly follow it, like says something to you that, you know, this might be a good idea or you should go execute this strategy. And I think people, you know, still tend to be responsible and accountable. And I think that there's this, there's this accountability that keeps people from doing too many crazy things with LLMs, I think we're in an okay shape with that when it comes to actually taking, like, quantity. Like when we talk about AI, like predictive modeling, there's a lot of really poor understanding of how these kinds of quantitative outputs can be used. And so, you know, as Things get automated, LLMs get more and more dangerous because they stop having a layer of sort of review on them. So that concerns me. But, you know, the, the marketing person who uses an LLM to sort of help them come up with a strategy, the one who's like, actually reading the output being like the, a good idea. And do I have evidence that supports that? That feels like a still very human thing to me. So I'm heartened by that.

[Mike Linton]

So as long as AI has some adult supervision, you, you would feel comfortable. Let, let's go back to the avocado analogy. You're a grocer. What are you teaching your students about other than these three, three points you made? What are some best practices our listeners can take away when they are. When they're trying to push data through the company? Because I think you mentioned the difference between data leadership and data science. Give our, Give our listeners some tips.

[Dr. Joel Shapiro]

So I actually talk a lot about data leadership. I just taught a new course on data leadership at Kellogg this last year, which is really interesting. Sort of the, the. I have like these seven big things that we talk about, but sort of most relevant to what you're talking about here is building trust is just incredibly important. And, you know, one of the most important things about building trust is expectations upfront. And so if we just talk about like something in The Euro grocer case with the errors in the data. One of the most important things that I do with anybody at the beginning of a data project is tell them that there will be errors. Data's always imperfect. It is always imperfect. And, you know, I don't think people lose trust when mistakes are made. I think people lose trust when no one talks about the mistakes. And more than anything else, I tell people that if you're going to be successful with data, you got to set the right expectations and you got to be transparent with things when they are imperfect. And that's just. That's just how to be influential, I think, generally. And how to get people to trust you generally. Super important with data.

[Mike Linton]

Got it. Any other tips you would throw down before we move on to the. The next topic?

[Dr. Joel Shapiro]

No, I mean, you know, you always want to know your audience, and you want to make sure that you know what they care about. Right. I work with a lot of data teams on improving their communication, and because they're not always the best communicators, as you could probably imagine, people who are really good with data, and the two things I always tell them are, you know, know exactly what the purpose of your communication is. Are you trying to sell somebody something? Are you trying to make yourself look like the smartest person in the room? Are you trying to influence a decision to know what you're trying to do and know what your audience cares about? And if you're really purposeful about that, a lot of communication issues can be solved for.

[Mike Linton]

I like it. We did a show with DJ Patel, big data scientist, and he said, you need a Spock on the bridge, but Spock needs to be able to talk to the crew, and the crew needs to be able to talk to Spock before anything happens.

[Dr. Joel Shapiro]

Yeah, yeah.

[Mike Linton]

And so I think that's really good. I know you have another case or maybe two that deal with asymmetrical risk. So let's talk about asymmetrical risk for our listeners and then give us the thumbnail and then we'll talk about, you know, case one and maybe case two.

[Dr. Joel Shapiro]

Yeah, sure. So the, you know, I told you before that my background is in public policy analysis. And. That's right. You know, sort of earned my stripes, so to speak, as an academic around more public policy issues before I shifted to more business stuff. And so anytime I see something that's more sort of in the policy space, my, you know, my radar always goes up and I'm interested. And so there's a super interesting and totally, totally sort of different context to it. In the Euro grocer case a number of years ago. So the state of Illinois, their department of Child and Family Services, had started being more data driven. And what they had done was they had started using a predictive model that was intended to help them predict kids at risk of bad outcomes, Specifically around, like, being abused or even perhaps being abused so badly that they might even die. Right. It was a risk tool, a risk detection tool that was supposed to help their caseworkers find kids who are more at risk than they might otherwise realize. So they could deploy. Deploy resources to the kids most in need of its resources. That's sort of the overall thing. It worked really well in other jurisdictions, but in Illinois, it completely tanks. And the reason it tanks is for two. Two things in particular. Number one, the model starts predicting that lots and lots and lots of kids, thousands of kids, are in, like, imminent danger.

[Mike Linton]

Yeah.

[Dr. Joel Shapiro]

Didn't seem realistic to anybody. But at the same time, there were these two toddlers, if you live in the Chicago area, like during the, like two 2010s, early 2000s, late 2010, something like that, two toddlers, tragic incidents of kids who are actually killed by their abusers. Just the worst possible case you can imagine. And when you go back to the predictive model, that model actually predicted that both of those kids were safe. It didn't have them as high risk, according to their prediction. And so the head of this department, the dcfs at the Department of Child and Family Services in the state of Illinois looks at this thing and she goes, it's predicting too much risk for some kids. It's predicting that these two kids weren't at risk and they ended up in these tragic circumstances. We're getting rid of the tool. It's not really helping us. And it's such a misread and a missed opportunity of how data can be helpful to us. And I'm happy to sort of wear that. The asymmetry.

[Mike Linton]

And the asymmetry, there is, look, a false. There's a false positive and a false negative, but the consequences of1 brings one terrible outcomes to the kids. But also, you know, the PR and the entire questioning of the model blows this up. So that's the asymmetry. Give us lessons learned from this thing. And what should you be doing if you're dealing with a super asymmetrical risk?

[Dr. Joel Shapiro]

Yeah. So look, I mean, when we start talking about data and doing predictions and stuff, sometimes you're right, sometimes you're wrong. We call those things when you're wrong. False positives or false negatives when you write they're true positives and true negatives. And a lot of times we get a little overly reliant on the technical folks to sort of build models that sort of optimize on these measurements. Business people need to know what the costs of being wrong and the benefits of being right are because there is asymmetry. In that child welfare case, a false positive means that you think a child's at risk when they're not. Maybe it's not so bad. You throw some resources at a kid who don't need it. Maybe it does get bad. Maybe you pull somebody out of a home when it wasn't merited. It can get bad. But man, the false negative is the worst. A child did protection and we didn't get to them.

[Mike Linton]

Yeah.

[Dr. Joel Shapiro]

Those costs, whether financial or human life or whatever else, those costs need to be known and managed by business leaders if they want to be successful, because they're the ones who need to sort of understand how the data influences the deployment of resources. And that's true if you're trying to save kids. It's true if you're trying to save customers. If you're trying to save customers, there's a much easier dollar figure to put on it. But that's the lesson is that decision makers, the business leaders, need to understand deeply the cost of being wrong, the benefits of being right, because otherwise that model is going to lead you to do things that might not make a whole lot of sense.

[Mike Linton]

Yeah. And that is, you're saying, look analytically, understand the asymmetry. Like, like if, if, you know, you should be weighting this super heavily towards one side because, because of the asymmetry. I know you just, in our pre. Talk, you were working on some other, other like asymmetrical risk, I think, or other thing along these lines in, in, in sports.

[Dr. Joel Shapiro]

Yeah.

[Mike Linton]

No, you're not done with this, but if you want to give us a preview, I'm, or our listeners would want to hear about it. So knock yourself out.

[Dr. Joel Shapiro]

Yeah, so, yeah, so I do a lot of work with professional sports teams, both on the business side, like the classic business stuff, marketing and sponsorship and digital engagement for fans and so forth. Sort of the fun stuff is the performance side and helping teams sort of build strong competitive rosters. And so part of what I've been doing is trying to understand how teams can plan for adversity. And adversity can be things like a player gets injured, an expensive player gets injured, or even like underperformance. And those kinds of things are predictable. And that's where the data and the AI come in. So if we can predict adversity, can we build our teams to be resilient in the face of an adversity? And this is something that I like to refer to as resilience by design. It's not just unexpected stuff comes up when we fight our way through it. That's a great kind of resilience. But can we plan for it? And the answer is probably yes. Like you said, I'm sort of right in the middle of this research here. But, you know, if you can imagine that there is a team in the NFL where I've been focusing most of my data, if you go most of my research, my analysis, if you have a player, a superstar, and they're getting a little bit older and your data tells you that they're going to drop off or they might get injured, sometimes teams will say, okay, that's it. We're going to trade them. We're not going to resign them, whatever it is. And what happens then if that player continues to be great? That's maybe a false positive, right? How you say. It could be a false negative, I suppose. But what happens if you gave up on this great player? The fans get upset, you missed on the opportunity to keep them and so forth. And the ways in which we talk about asymmetrical risk, false positives and false negatives, when it comes to bringing talent onto your team and becoming competitive and remaining competitive, it's really interesting when we start thinking about it in sports. One of the fun things about sports is everything is so easily measurable.

[Mike Linton]

Oh, yeah.

[Dr. Joel Shapiro]

Productivity of a team is, did you win the game? Did you make the playoffs? And so this notion of can we predict adversity and be resilient to adversity really becomes interesting. And it's fun because I'm a sports fan.

[Mike Linton]

All right, so if we come to the show, if you are willing to share this with our listeners when it's, when it's done.

[Dr. Joel Shapiro]

Yeah, totally.

[Mike Linton]

That would be super fun. So. All right, so we're running towards the end of the show. Before we get to our last question, our famous last question, or we like to think it's famous, anyways, what should our listeners take away from all this? Just from, from your chair you're talking to, to all these business folks out there, what would you, what would you tell them?

[Dr. Joel Shapiro]

I think you know that one of the most important takeaways from all of this is what, what everybody knows. But it's going to be a reminder, which is that, you know, data doesn't make decisions, Data doesn't make decisions, People make decisions. And, you know, data needs to be helpful to them. And that means it needs to be meaningful, it needs to be actionable, but it also needs to be presented to them in ways that, you know, get them to buy in and trust. And, you know, there's so much more to success with data today than just the technical components to it. And I know people know that, but I think sometimes it gets a little bit underappreciated by both the data teams and the business teams.

[Mike Linton]

I think that's a super rep on the show. Which brings us to our traditional last question. You have to take. You could take both parts of this or one, but you have to take at least one. Okay, so practical advice for our audience we haven't discussed yet and. Or the funniest story you can share on the air.

[Dr. Joel Shapiro]

I find it very hard to be funny on demand. I'm going to share just a quick snippet here. When I was doing a big consulting gig with a big financial services firm, helping them sort of become more data driven and all this kind of stuff, this big training program getting ready to go out in front of this crowd, all their employees were super excited and jazzed up about this new analytics program. And one of the corporate sponsors, one of the big wigs at the company pulls me aside before I go out and he says to me, I'm glad you're here. I'm glad we're doing this program. The data team is pissing me off. That was a weird thing to tell me. I'd never met this guy before. I think highly of their data team. I'm like, what's going on? He said, you know what? I know my business really well. I know what I'm doing. And they keep trying to tell us what to do. God. And that just stuck with me. That just resonated with me that if we in the data space continue to be seen as telling smart and accomplished people what to do differently, we're all kind of sunk. It's just not the way to make change in an organization. And just the way he said it, with sort of such, you know, ferocity, they keep trying to tell me what to do. And so that just was a big moment in my career about thinking about making sure you had data solves problems, but making sure that we know how to work with people to make sure that we are supporting them and not trying to tell them what to do differently. Because there's just almost no way that that's going to work. So the practical advice is for the business teams maybe be a little bit more tolerant to the data teams who think they've got the one right answer for the data teams to be really good at thinking more deliberately about how to influence business people to do what the data might suggest is a great idea.

[Mike Linton]

I think that is a great wrap. And the lesson that I have always taken away is the difference is whoever is making the decision owns the consequence and the data team usually doesn't own the consequence, which means the data team has to be aware of, of, of how that leader is leading and the leader has to trust the data team enough to take some risk with them. So yeah, I love that.

[Dr. Joel Shapiro]

Great way of putting it.

[Mike Linton]

I think that is a great way to wrap the show. Thank you so much, Joel. We'll see you later. And thanks to everyone for listening to CMO Confidential. If you're enjoying the show, hit the like button and subscribe. And look for all of our shows on Spotify, Apple, YouTube and the I Hear Everything network which include the rise and fall of peloton as seen through the lens of cltv, the Budweiser case, how not to manage socio political issues, the Warby Parker case. I can see clearly now with my CLTV glasses on. And is artificial intelligence like taking the red pill or the blue pill? Hey all you marketers stay safe out there. This is Mike Clinton signing off for CMO Confidential.