← The Rational Reminder Podcast
The Rational Reminder Podcast
Episode 361: Alex Edmans - Finding "The Truth" in Economics, Finance, and Life
Loading summary
Transcript76 lines
- [00:00]
A
Foreign.
- [00:04]
B
This is the Rational Reminder podcast, a weekly reality check on sensible investing and financial decision making from one Canadian. We are hosted by me, Benjamin Felix, Chief Investment Officer at PWL Capital. All right, this is my first time doing an intro and hosting a guest by myself. Cameron's done a couple like that, but this is my first time I had a great conversation with Alex Edmonds, who is his second time on this podcast. He is a professor of finance at London Business School. He's got a PhD from MIT as a Fulbright Scholar. He's previously a tenured professor at Wharton and an investment banker at Morgan Stanley. He's done a lot of great work on what he calls growing the pie related to social responsibility. He's done a couple of extremely popular TED talks and his previous book, Grow the Pie is what we talked to him about last time he was on the podcast. And his latest book may contain how stories, statistics and studies exploit our biases and what we can do about it, which is published in 2024 is what we discussed in this episode. It's a great book and I think we had a great conversation covering the topics. It's thinking about how finding the truth today, because there's so much information available to people is increasingly difficult. And Alex talks through this concept that he's come up with called the ladder of misinference, where he goes through different types of information and how they can be misrepresented or misinterpreted in a way that makes the truth very difficult to determine. So Alex talks through the different rungs on that letter of misinference and what people can do to look for the truth. So that's it. That's what we talked about. There are a couple really interesting examples of very well known statistics like the 10,000 hour rule and the importance of sleep that turn out to be, when you dig into the actual sources of those statistics, maybe less true than I think a lot of people think. In some cases the data have been misrepresented and in other cases the data to support the claim just weren't there to begin with. And so we talked through different versions of that type of concept going through increasing levels of strength of evidence. And so from a statement to a fact and from a fact to data and from data to evidence and evidence to proof and why each rung below the one above it is not the thing above it. So for example, a statement is not a fact and Alex explains why and a fact is not data and so on and so forth. That's the premise of the book and that's the premise of the conversation. I think it's really important. It's basically how to think like a scientist and how to combat confirmation bias, which is very problematic. People are very eager to find evidence. And there's some research that Alex talks about, about what happens in our brain when we find confirming evidence. People really want that, which is problematic. And he also talks about black and white thinking. It can make things seem very compelling when you say this is true and this is false, or things are this way and not that way, when in reality the world is quite gray. I think that's enough for an introduction. So we'll go ahead to our conversation with Alex Edmonds. Alex Edmonds, welcome back to the Rational Reminder podcast.
- [03:30]
A
Thanks Ben. It's great to be back.
- [03:32]
B
Super excited to be talking to you. Your new book is excellent. I really enjoyed reading it.
- [03:36]
A
Thanks. Well, after you sink two years into writing something, I always appreciate it when people like what came out.
- [03:41]
B
Really important stuff as listeners are about to find out. To kind of set the tone, can you tell us the story about the erroneous evidence that you watched influence the creation of a law in the UK?
- [03:51]
A
Certainly. So in 2016 there was a UK inquiry into corporate governance, which is the way that companies are running. And being a professor who studies corporate governance, I submitted some written evidence. In response, they invited me to give all evidence in front of the UK House of Commons. It's in front of a select committee. And so I was grilled with some questions. But before my session there was a session beforehand which I sat in on just to make sure I knew what was going on. And the witness before me mentioned some research which sounded quite noteworthy. So she said that there was evidence which suggested that the lower the gap between the CEO's pay and the worker pay, the better the company's performance. And this was really interesting to me because some of my own research is about the merits of treating employees fairly. So I thought this would be another bow to my arrow, so I might just tweet it from the rooftops and share it with other people. So I wanted to find out this research, I could share it. And I looked up the study because it was in the witnesses written submission and it said completely the opposite. It said the lower the pay gap, the worse the performance rather than the better. And I thought, well yeah, I am pretty nervous before my own session but clearly I should know how to read. And it was there as clear as say. And I realised what had happened is the witness had quoted a half finished version of the study. So I was looking at the finished version. The finished version showed the lower the pay gap, the worse the performance. But the witness quoted the half finished version because it gave what it wanted to be true. The witness was from the trade union with a strong public position against pay gaps. And so it wanted to quote evidence finding that pay gaps are bad for performance.
- [05:34]
B
That's wild. That made its way through into the law, though. How did the rest of the process work?
- [05:39]
A
So what happens is that there's lots of witnesses who give evidence at the Select Committee. You would not necessarily make law based on one particular witness statement. And indeed, after finding this out, I reported this to the clerk of the Select Committee. He said, oh, this is not good. You should write some more evidence in. We will publish it. And they did. And yet they ignored my rebuttal. It's not so much a rebuttal, it's a clarification. They still went ahead by quoting the original evidence as if it was gospel. And this was really alarming to me because, okay, you think, yes, there's misinformation. Maybe somebody on Breitbart News or an Instagram influencer will spread misinformation or. But the UK House of Commons Select Committee, for something that might eventually become law, was quoting something they knew to be wrong. So sometimes we're misinformed because we can't look at every single information out there. But I had informed them, I had sent my clarification, they had published it. Yet this was something they didn't want to be true. Why? Because the UK at the time there was quite a lot of outrage against high cepay. That was the reason for having this inquiry to begin with. So they just latched onto evidence they knew was incorrect. And eventually it now became law that every company in the UK above a certain threshold has to publish pay ratios. Now, the point here is not about pay ratios being good or bad. Different people might have different opinions, but just how careful we need to be about evidence. And even evidence from a supposedly trusted source like a UK government report may well not be gospel.
- [07:12]
B
That's pretty scary stuff. What does it mean to say that we live in a post truth world?
- [07:17]
A
It is that there's lots of misinformation out there. And you might think, well, this is obvious. I knew this before listening to this podcast. But what I want to highlight is misinformation is a bigger problem than what we think. So we think that misinformation might be an outright lie. So if somebody claims that Barack Obama is not a Natural born US citizen. We can falsify that by showing his passport. But sometimes something could be truthful, but it could still be misleading. So indeed it was true that one version of that paper had found that low pay gaps are linked to better performance. And they indeed quoted that half finished version of the paper. The witness cannot be prosecuted for spreading misinformation because what they said was strictly true. That earlier version did find the result, but they had forgotten or they completely had failed to recognise that there was a later version, a finished version, after going through peer review and correcting its mistakes, which found the opposite. So we can be misled even by truth. So close truth does not mean just outright lies. It might be things that are truthful but are misleading because they've been superseded.
- [08:28]
B
It shows up in that story that you just told. Can you talk more generally about the role that confirmation bias plays in making it difficult for people to find the truth?
- [08:36]
A
So you might think, well, misinformation that just happens to uninformed people. I am a smart listener to this podcast. I'm rational and therefore I won't fall for this misinformation. But what I'm highlighting is that many people, including me, make these mistakes. In the book, I go through examples where I myself made mistakes. It's not that we're bad people, it's that we're people. And if we're people, we have biases. And one of the biases, as you mentioned, Ben, is confirmation bias. So that's the idea that we have a view of the world and if we see something that supports that viewpoint, we lap it up uncritically. So maybe the trade union had a view that high pay gaps are bad, low pay gaps are good. As soon as they saw a paper that supports that which confirmed their viewpoint, they lapped it up. They didn't even bother to look at whether that paper was updated even though it was 2016. They looked at a 2010 version. The finished version that I looked at was in 2013, so it'd been out for three years, but there was no need to look for it if you already have the answer that you want.
- [09:41]
B
What should people do to combat confirmation bias when they're making a decision themselves?
- [09:47]
A
I think it's like any sort of addiction, which is to recognize that you have the addiction to begin with. So the first step towards alcohol recovery is to know that you have a problem. Now, the listeners might think, well, that's a pretty strong statement to compare bias to an addiction, but I don't think it's strong because it has the same roots, and those roots are psychological. We are actually addicted to seeing things that confirm what we'd like to be true. There have been neurological studies which show that when you see something that confirms your viewpoint, it releases dopamine in your body. It feels as good as finishing a long run or having a laugh with friends. Yet when we see something we don't like, it triggers the amygdala. And that is the part of the brain that induces a fight or flight response. It's the part that's triggered when we're attacked by a tiger. Just to recognize that we naturally have a preference towards listening to things that we will like and shutting out things that we don't, then we will need to actively combat this by going out and reading things that we won't agree with. So this might be reading articles or from a newspaper with a different political viewpoint. It may well be in a meeting, encouraging people to disagree with me, giving time and space for people to share dissenting opinions. Any active action that we can take, having recognized that we will naturally gravitate only towards confirming viewpoints, is a way of getting more informed.
- [11:14]
B
You have a toolkit at the end of the book. One of the things that you wrote there that stuck with me was to ask yourself if you want the thing that you're looking at to be true as a filter to see whether you may be affected by confirmation bias.
- [11:27]
A
I think this is important because I'm trying to lead the reader to smarter thinking, but I need to recognize that the reader doesn't have the time to check every single footnote in every single paper. So where can we be selective and apply the 8020 principle? If there is a study or a claim on something that's number one, important and number two, that we really want to be true, then we need to be particularly careful. Why? Because, number one, we might not scrutinise it, but number two, other people haven't scrutinized it. Why we might have heard of that study is it's been round our echo chamber. Other people have liked it and shared it without necessarily checking it just because they like the punchline.
- [12:06]
B
Empirically, how severe are the effects of confirmation bias?
- [12:10]
A
They're very severe. So what studies have done is in addition to looking at how this affects the brain is the neurological basis is they've looked at how it affects outcomes, and perhaps one really serious outcome is false conviction. There were studies of the most serious investigative failures, and what the study looked at is what were the various causes of this maybe the DNA evidence was incorrect, but the clear winner by some margin was confirmation bias. So what does this mean? In a criminal investigation setting, you might have a hunch about who is guilty. Maybe that's somebody with a particular appearance. Sadly, it can be somebody with a particular ethnicity or other demographic characteristics, or they may have visible markings like tattoos, which you use as a stereotype and then every piece of evidence you interpret as consistent with that suspect. But what I highlight in the book, in all settings, not just criminal settings, but with evidence, is that there's multiple explanations for a funding. So if I find more sustainable companies perform better, a sustainability advocate like me will claim its sustainability causes better performance. But alternative explanations is that better performance could cause sustainability. And similarly, in a criminal investigation, it may well be that there's a forensic evidence that the killer was a 5 foot 10 male. Sorry, not forensic evidence. Eyewitness evidence that the killer was a 5 foot 10 white male. And that fits in your suspect, but it could also fit other people who were in the facility. They had motive, means and opportunity. But you don't explore those alternative suspects because you're just narrowly framing on your particular culprit.
- [13:52]
B
Something that I've definitely noticed is that on divisive issues people will generally agree on the facts, but they may disagree severely on the way to interpret the facts.
- [14:02]
A
Absolutely. That's very selective interpretation. So if there is a study which finds that sustainability doesn't pay off, then on LinkedIn there's no shortage of comments as to this is one study out of many. Or maybe this was just in a particular time period. And yet when there is a study finding sustainability does pay off, then people will just shout this from the rooftops without considering the fact. This is just one study. We know how to be discerning and critical, but we are doing this very selectively. We criticise things that we don't like. But what I'm highlighting is we need to apply the same discernment to things that we do like. And why I think this is quite realistic for even busy people to do is what this highlights is that the skills for discernment are already within us. So the solution to misinformation is for everybody to get a PhD in statistics. That's clearly unrealistic. I don't even have one myself. But if instead we already know not to overweight weight one single study, we know that correlation is not causation. We know that a cherry picked anecdote may not be representative. Apply those same checks when we see something that we do like how impactful.
- [15:15]
B
Is biased search as opposed to biased interpretation?
- [15:19]
A
What we've talked about so far, Ben, is one aspect of confirmation bias, which is I call biased interpretation. Once we get information, we interpret it in a biased manner. So if it's something we like, we immediately will accept it. If it's something we don't like, we will either not read it or if we read it, we read it, trying to pull it apart. But there's a second form of confirmation bias which is, I think, just as important. It's biased search. Do we bother going out and looking for different viewpoints? This even affects things like Google searches. So if I want an excuse to drink loads of red wine this evening, I could search for why red wine is good for your health. I'm sure I'd find a ton of studies showing that red wine leads to longevity. Now, how I looked at the opposite, how I googled why red wine is bad for your health, then I could have found some opposite studies. But because I framed the question in a particular way, I'm only going to get a particular answer. And this applies much more widely than just Google searches. It applies to what newspapers we might subscribe to or read. It applies to who we follow on social media X or LinkedIn. We often will be in echo chambers. We're just looking for information that confirms our viewpoint. Sometimes biasearch is even more serious and that even if we don't explicitly say something, we will receive selective information. So what do I mean by this? So let's say you're a boss at work. Often people think, I will want to placate the boss. I will want to give information that he or she will agree with, even if I say nothing to suggest that I'm biased. Most people assume that you only want to see something which is confirming. So this is why, again, we need to be intentional. As a boss, say, I would like you to challenge me and show me why I'm wrong. That is important to ensure that you see the other viewpoint. This actually happened to me yesterday. So yesterday I was teaching in class and I always encourage students to challenge me. And there was a question I asked and the student gave an answer and I gave a counterpoint, a different viewpoint to that student. She actually hadn't taken my core class, so she hadn't known that I like to have debates in different viewpoints. And then she came up to me in the break and was like, really worried. And she said, oh, I sent you an email just apologizing for having a different viewpoint from you. And challenging you. I really would like your reference to a PhD program. I'd like to study a PhD. And I said, oh, there's absolutely nothing wrong with challenging me. Actually, if you want to do a PhD, I would like somebody with independent thought. But because she is from a culture where a professor is somebody who's infallible, who's an authority, it was actually not natural to her to know that she is able to challenge me. So that is something that perhaps I should have recognized myself rather than assuming that everybody knows that I'm open to challenge. Maybe I should have made that more explicit at the start, particularly for people who, who hadn't taken my core class.
- [18:19]
B
Something that I've noticed and I have no data to support this, but just my intuition is that the Google search example has become even worse with generative AI, with ChatGPT, this is actually fair.
- [18:30]
A
So people always think, well, isn't AI the solution to this? Because AI is unbiased, it's going to look for everything out there. But I have used AI myself and I've used it just to do research. I haven't deliberately tried to skew it one way. But when I ask what is the evidence that diversity improves performance? Why? Because I'm doing a study on diversity myself, it comes up with some very flimsy studies by consultancies suggesting the evidence is unambiguous. And then when I write back to it, I say none of these studies were published in top scientific journals. Can you give me the scientific research it finds completely the opposite and it admits it made a mistake. But had I not sent that follow up challenge, had I been somebody who wants to believe that diversity pays off, I would have accepted the first thing uncritically. So even something like generative AI, whether it's programmed to make you like it and want to continue interacting with it, and if that is something as part of the algorithm to keep you there, then it will give you stuff that you want to be true.
- [19:34]
B
How does someone's level of knowledge affect their susceptibility to confirmation bias?
- [19:39]
A
What you would hope or you might think is the more knowledgeable you are, the less susceptible you are to confirmation bias because you're able to think more rationally. You know the difference between correlation and causation. But the issue here is not the lack of knowledge, but it's the inability or unwillingness to use your knowledge because of your biases. And for that reason, actually knowledge doesn't help because the reason why we're not interpreting information correctly is not through Lack of knowledge, but our biases. In fact, actually knowledge can make things worse. Why? If we see something we don't want to be true, the more knowledgeable people can come up with some reason to dismiss this. And one very salient example was Deepwater Horizon. So this was the explosion of one of BP's least rigs in the Gulf of Mexico. And so what happened was you need to do a test, known as a negative pressure test, to check that the oil rig is safe before removing it. And they did the test three times and this always failed. And they came up with an explanation called the bladder effect, which is pretty technical. I describe it more in the book as a reason for why this test was given the wrong result. Because in their eyes the rig was safe, so they should remove it. And so they came up with an excuse to do a different test. That test passed, and then the explosion happened. And afterwards there was an official investigation into Deepwater Horizon which found that this bladder effect was a complete fiction. Nobody in their right mind would have come up with that as an excuse for the failed test. But if you're really smart, then you might convince yourself by coming up with something convoluted.
- [21:24]
B
That's a crazy example. What role does black and white thinking play in making the truth elusive?
- [21:30]
A
So far we focused on one bias, which is confirmation bias. And that applies when we have a pre existing view. We latch onto stuff that confirms that view and we reject stuff that contradicts it. But what happens if we don't have a pre existing view? There might be certain topics that we're neutral on. And this is indeed where black and white thinking comes into play. Which means that even if we don't have a pre existing view, we have a tendency to see things in black and white terms. Something could either be always good or always bad. We have a binary opinion. So one example might be food with protein. Most people do have a view. Protein sounds good, it builds muscle. With fats. Most people think fat makes you fat, that's why it's called fat. But with carbs, most people may well be neutral. But the Atkins diet, it preyed on black and white thinking by suggesting carbs are always bad in all measures and all types of carbs. Not just complex carbs, but simple carbs as well, and so on. And so why was that such a successful diet? Despite extremely weak scientific evidence, it played on black and white thinking. It was really easy to follow. You don't need to track your calories and make sure that carbs are no greater than 30% of your daily calories. You just said, let me avoid carbs. And this is all carbs. It wasn't just refined sugar, it was complex carbs as well. Notice had Atkins had completely the opposite diet, which is eat as many carbs as possible, he may well have been just as successful because that's just as easy to follow and it plays into black and white thinking. We see the same idea with some superfoods right now. So what this means is to Pennabastella, Atkins did not need to be right, he just needed to be extreme.
- [23:19]
B
The way you tell that story in the book is very well told, but it's also very scary. What are the problems that can arise from black and white thinking?
- [23:27]
A
I think it means that we just ignore the nuances and shades of gray. We think that something is always good or always bad, when even if something is good, then it could still be good only up to a point. One quite salient example is some people might die because of black and white thinking. So when you run a marathon, people tell you you need to hydrate little and often, but drink as often as possible. And there have been marathon runners who've died from water intoxication, where they drink so much that essential minerals are diluted to fatal levels. So even though water is good, it's not always the case that more is always better. The same is true with management practices. You might think, oh, let's give employees more autonomy, which seems to make sense. We want to delegate, but too much autonomy might mean that they're not coordinating their efforts, they don't have direction. Certainly providing employees feedback is said to be a good thing, but too much feedback too often will lead to short termism rather than focusing on long term goals.
- [24:27]
B
So basically, there's a lot of gray in the world, which makes black and white thinking pretty challenging.
- [24:31]
A
People might think, oh yes, I knew that there were shades of gray. I didn't need the rebook to do this. But when I talk about shades of grey, shades of grey may come in many different forms. So one form that we've just talked about is the idea that more might not always be better or less might not always be worse, and so on. But another type of shades of gray is even if something is good in general, not everything within that umbrella may well be good. So let's say if sustainability is generally good for financial performance, not everything under the sustainable bucket may be good. So let's say employee welfare is good for performance, which makes sense because it leads to motivated workers. But having a Catholic values Screen might not be positively correlated with financial performance. But if we view the world in black and white terms, then anything which is called sustainable, we think is good. Similarly, anything which is called cholesterol might be bad, when actually there is good cholesterol and there's bad cholesterol. But another form of black and white thinking is we see the world in labels and we treat everything under that label the same.
- [25:40]
B
In the book, you kind of set everything up by going through confirmation bias, black and white thinking. Then you get to the ladder of misinference, which is so cool. Can you describe what the ladder of misinference is?
- [25:50]
A
So why did I come up with this to begin with? It's before I wrote my book, I read other books about misinformation and they were good. I'm not going to attack them because I view them as rival books, but they were so good and they were so comprehensive that they came up with the 263 ways in which we can be misinformed. And then it's difficult for a reader to remember all 263 and to put them into practice. I wanted to boil down all the types of misinformation into just four categories so that it's easy to put into practice. And those four I display in a graphic called the Ladder of Misinference. So why a ladder? Well, when we start from some statements and we draw some conclusions from them, we are climbing a ladder. But why I call it the ladder of misinference, which is obviously a play on the ladder of inference, is that sometimes the steps that we make when we draw conclusions from raw data are incorrect. And so let me just talk through for now. So the first step is a statement is not fact. It may not be accurate. So what do I mean by this is we quote statements all the time as if they were the gospel truth. Culture eats strategy for breakfast. Peter Drucker said this, so it needs to be true because he was a management guru, but it may not be the gospel truth. And indeed, in that case, there's two issues at least. The first is that Peter Drucker never said that, and so this is a misattributed quote. And second, and more seriously, even if he had said that, it doesn't mean that it's true, because where is the evidence that culture always beats strategy? Yes, he is a famous management guru, but that doesn't mean that everything that comes out of his mouth is correct. There's a lot of famous people. I'm sure the listener can think about who says things which are not correct. So what we'd actually want to look at is is there evidence behind that statement? Has there been a careful study of, of the link between culture and strategy and performance? And even if there isn't hard evidence, this doesn't mean that what he says is definitely incorrect. But rather than Peter Drucker states, proves that culture eats strategy for breakfast, we will say, well, in his subjective opinion, culture is more important than strategy. We will still put some weight on that because he's a guru. But this doesn't mean that strategy is irrelevant, which is how some people have interpreted that black and white statement.
- [28:18]
B
Some of the examples you have of statements that are just not true but have persisted for a long time are mind blowing. There's actually one I don't think was in your book that I was reading about recently about how lottery winners tend to go bankrupt a large proportion of them. That's another one of these that there's no basis for that. Somebody said it once and it was attributed to a study that was never done, but now people believe it to be true. And you've got a bunch of examples like this in the book. Can you talk about where the famous 10,000 hour rule for mastery comes from?
- [28:45]
A
This was from a best selling book by Malcolm Gladwell called Outliers. And the popular version of the rule is 10,000 hours is the key to success. You can be successful in anything that you want to as long as you invest 10,000 hours into studying or practicing it. And immediately this plays on confirmation advice. We want this to be true and we think it's true because we're taught from a young age that practice makes perfect. You can do anything that you set your mind to. And this is empowering. So this suggests that the world is your oyster. You don't need to be limited by your genetics or your past background. And I had to admit in the book that I myself had taught this to my students at Wharton. Why I'm a finance professor. What has this got to do in finance? Well, at the end of my final lecture, because I start to teaching right at the start of the mba, then my final half an hour at the end of my final class is looking forward for the rest of the two years and I give some advice. And one of them was the 10,000 hours rule, which is that to do an MBA, you can push yourself outside your comfort zone. You can redefine yourself. Maybe historically you always worked on quantitative factors, but now you can become a leader, a master negotiator and public speaker as long as you put your time in. And when I said that, I would get a lot of nods, people would say, oh, that was an inspirational, empowering speech. But then later on, I found that the evidence for this was really, really weak. So Gladwell claims that this rule applies in every context. Yet the evidence he cites is limited to just violin playing. And what works in violin playing may well not work in public speaking or neurosurgery. But even worse, that study never measured 10,000 hours and it never measured success. Which is a bit of a problem if you think that 10,000 hours is the secret to success. Well, first, how did it not measure hours? So all it did is it asked people to report the hours that they had practiced since age 5. Now, these violinists were currently 23 years old, so they had to remember from age 5, 18 years ago how much you practiced. You might well not remember how much you practised last week. Here is the problem. If you were successful right now, you might well say, yeah, I probably did practice a lot to get here. I had to work hard. You don't want to say I was just born with it and was lucky. And conversely, if you're not successful, you won't say, I practised a lot when I was young, because then you'd have to admit that this was all a waste of time. Leads to higher reported hours rather than hours leading to success. These are just self reports. But another issue is it never measures success, not in any objective manner. It just asks the teachers at this violin academy who has the greatest potential to, in the future, play at a good orchestra. And here it may well be that, let's say the teachers knew how much the students practiced. Probably if you know a student practices a lot, you will probably say, yeah, I think he or she will be successful. If somebody's goofing off and not showing up, you'll probably say that they will be unsuccessful. So for something which claims to study the link between practice and success, it doesn't measure actual practice, but memories of practice 18 years ago. It doesn't match an actual success, but a subjective evaluation, that's wild.
- [32:18]
B
And that statistic is so commonly cited. There's another one that was pretty striking. There's a book that a lot of people have read, why We Sleep. Can you talk about the example from that book?
- [32:28]
A
Yeah. So this book, why We Sleep, by Dr. Matthew Walker, because he stresses on the front cover his scientific credentials, argues that sleep is the secret to success. We're in a national or global sleep epidemic. People are not sleeping enough. This confirms what we want to be true. We want an excuse to sleep more because it's going to lead to greater physical health, greater rate of mental alertness. And we would like to believe that our over eager beaver colleagues who wake up at 5am they will get their coffins. And what might their comeuppance be? Well, maybe if they're not fully alert, they might just have accidents. And indeed there is a graph which shows the more you sleep, the fewer the accidents. If you sleep for nine hours, you have fewer than eight, that's fewer than seven, fewer than six. Yet if you go to the underlying paper behind this, he actually shows a quite different graph. It has another bar for five hours of sleep. And those people who sleep for five hours have fewer injuries than six. They also have fewer injuries than seven. But because Matthew Walker doesn't like that, he cuts this out of the graph. He just shows something which is really selective. You think that this is a book founded on lots of data and evidence. Again, it's by this Dr. Matthew Walker who goes out of the way to stress his credentials. And yet there's a lot of errors in this. And this is only one of many errors in the book. There's an entire blog post by Alexei Guzzi on why the book why We Sleep is riddled with scientific errors. And so again, even if something is said to be based on data and science, you could just be quoting selectively, going so far as to cut out a bar of a graph because it doesn't support your viewpoint. That is just like a police officer hiding some evidence because it exonerates his suspect.
- [34:13]
B
Yeah, that one's crazy. Because you show the chart from why we Sleep and then you show the chart from the paper in the why We Sleep. It shows this monotonic relationship which is completely destroyed by the actual chart from the paper, which is like you said, it's like hiding evidence. How is something that egregious not flagged by an editor or a fact checker?
- [34:31]
A
It's because people just may not want it to be true. If you are an editor, you might think this is a guy with strong scientific credentials. So another thing is Matthew Walker misrepresents his scientific credentials. So he said he had a PhD from the prestigious Medical Research Council of the UK. There's no such council which ever gives any degree. So this gives you grants, it doesn't give you a PhD. He's got his PhD, I think from the University of Newcastle. That's an okay university. It's not Oxford or Cambridge or London and so on. So that is also misrepresented. But people might think, okay, this is a guy who's got great credentials. He's quoted a paper. If you quote something you don't have the time to go through and look up every evidence, you might trust him. But I think what is even more surprising or more disappointing is even after all of these issues have been raised on this very long article critiquing his book, nobody's really changed their opinion because. Because I say, okay, well, despite all of that, oh, it's my gut feel which thinks that sleep is still important, or my lived personal experience suggests the more I sleep, the more alert that I'm going to be. But again, this is sort of mixing up correlation and causation. Right. Why are you able to sleep a lot? It's probably that lots of other things in your life are going well. In the areas I'm sleeping more, then I'm probably going to be happier. But because lack of stress means I'm going to sleep more in the period in which I am not sleeping, it's because I'm having to juggle loads of other things right now. I just had another child a month ago and maybe that is going to cause a lot of stress and a lot of busyness, not necessarily the lack of sleep. Similarly, this might happen with political outcomes. So in the Brexit referendum, where the UK voted to leave the European Union, there was an incorrect claim posted on the side of a bus that the European union cost the UK 350 million pounds per year. That was a sum which was completely exaggerated. When that was corrected, people said, oh, well, that never influenced my viewpoint anyway, I would have voted the same way. But you would say that because you don't want to admit that you were misled by something. So once you have a particular opinion, even after there is some. Some disconfirming evidence, people are not changing their view. So this is the often quoted phrase that a lie has gone halfway around the world before the truth has laced up his or her shoes is once something has gone viral, people stick to it, even if there's disconfirming evidence.
- [37:07]
B
Sleep's a tough one. I do like to sleep. How should people verify whether a statement is actually based on a fact?
- [37:15]
A
One thing is actually just to try to search for. This is trying to search for disconfirming evidence. So if you were to look at why we sleep errors, or you might look at this in any type of popular book, 10,000 rule critique, then you might find this. And so while I've talked about the dangers of getting information from the Internet because there might be less trusted sources. On the other hand, what it does mean is we do have access to a wide source of information. And even if somebody who might not be a massive household lame like Alexei Guzzi, even if he's a serious researcher and has done his homework, this is something that it will come out with. So try to find something which is disconfirming. Sometimes it may well be as simple as just going to the underlying article. Now, I don't expect people to go through every single word of a scientific paper on sleep, but just to look at that picture, you can just see her. That picture is very different from the picture that always ended up being quoted.
- [38:11]
B
Those are examples of how a statement may not actually be based on a fact once it's been verified. How is a fact not data?
- [38:18]
A
This is the second step of the ladder of misinference. So you might think, okay, what Alex has said is a statement is not fact. Let's check the facts. Now, if we know that something is a fact, are we home and dry? Well, the answer is no, because even if something is a hundred percent true fact, it could be cherry picked. It might be the exception that does not prove the rule. So one example is Simon Sinek's book, Start with Y, which claims that having a why is the secret to success? And he gives a number of facts to back this up. For example, Apple, starting with Y. And Apple is extremely successful. There's no denying that. And he also gives other facts, other anecdotes. For example, it applies in the nonprofit sector, Wikipedia, extremely successful. It applies to individuals. So the Wright brothers, they were the first to have a test powered flight. But all of these things could be cherry picked. So they could well be hundreds of other companies or nonprofits or individuals that also started with Y and they failed. But we never hear about them because Simon Sinek will never tell you about this. So this is the difference between fact and data. So facts are individual stories. Data is something which is large scale. And the issue here is not just authors like Simon Sinek. The issue could be business school professors. So I'm saying this as much to myself as I am to the reader, is that in business schools we like to teach with case studies. But how do you choose a case study? You choose the most vivid and striking example of, of the particular point that you want to show. So it may well be a company which was sustainable in, a pioneer in that and had massive performance, whereas there might be others which are sustainable companies who Failed. So I do use stories and I do use case studies in my teaching and in my books. But when I do use them, I always try to make sure that it's backed up by large scale evidence. So for me, I start with the large scale evidence and I find a story which brings that evidence to light. And then sequentially, yes, I lead with the story first and then go to the evidence to back it up immediately afterwards. But often what people will do will just find a story which supports that viewpoint, find other examples that support that viewpoint as well, but they won't do a systematic study of the large scale evidence.
- [40:43]
B
What are the steps involved with separating fact from data?
- [40:47]
A
What you'd want is something similar to a randomized control trial in medicine. So what happens there? So you would take a number of patients and they're all suffering from some affliction. You give some of them a treatment, a drug or a behavioral intervention, see how many get better and how many get worse and give others a placebo and see how many get better and how many get worse. And you compare the two. But importantly, your data set needs to contain people who took the drug and still got worse. That's companies that started with Y and yet they fail. But they will never be in any Simon Sinek book. And similarly, you need companies that did not. Sorry, you need people who took the placebo and yet they still got better. That's like companies that did not start with Y and yet they still succeeded. But again, you don't see this in any Simon Sinek book. So he will only show you the companies that started with Y and succeeded. He did. And even if Those examples are 100% fact, it's what he is not telling you, which is where all the action lies. All of these other companies that started with Y and failed, and all of these companies that succeeded without Y. And why is that so striking? It suggests that misinformation is really hard to police. So some people might think, oh, why is it not the government stepping in and prosecuting it? To one of your earlier questions, Ben, why don't editors come out and check this? But the issue is not what you publish and what you disclose being incorrect. Maybe what you're saying is correct, but it's what you're hiding, which is where all the action lies. And it's really difficult to prosecute Simon Sinek for not telling you about all of the counterexamples. He could claim he just never knew of those counterexamples, perhaps because he engaged in bias.
- [42:42]
B
Search the Simon Sinek example touches on this. But what role do narratives play in how people interpret or misinterpret facts?
- [42:49]
A
Substantial. So this is the idea of the narrative fallacy, which is, even if there is no relationship between two factors, if you can weave in a compelling story, that story is memorable. And then we perceive a link when there is no link. And again, this is deeply ingrained with us because even before we had writing, it would be the oral tradition. People passed in information through stories. So the narrative that Simon Sinek weaves in is, well, if you have a why, people will buy into this. It inspires people. People buy Apple products not because of what they do, but because of why they do it. They buy it because Apple has this vision and this purpose. Now, if you were just to look at that statement with a clear head, that sounds crazy. That's a load of rubbish. Why do you buy Apple products? Because of their functionality, because they work, because they have the apps, because of the reviews, because other people will have iPhones. They work. I don't really know why Apple did this, nor did I care. And I don't think this is unique for me. I think most people will buy products because of their reviews. Other people have them, their connectivity, their functionality. But the idea that it's a why, it's a vision, that sounds more compelling, that might say, as a leader, I don't need to get involved in the boring stuff, making sure that all the wiring is correct and everything works. I can just create a vision. And that's something that many people think is much more compelling.
- [44:23]
B
Yeah, it makes sense. Stories are very powerful. So as we move up the ladder of misinference, we start with a statement, then we go up to a fact, and then we go up to data. Once we have data, why is data not evidence?
- [44:36]
A
If the solution is not to look at a select example, then why don't we get data, which is the full sum, the full picture, the companies that start with Y and the companies that did not. Well, the issue here is that you may have a strong correlation, but it's not causation. There could be multiple explanations for the finding. So let's say in general, what we find is that companies with a purpose statement, a why, do tend to be successful. And we've done this. We found this link looking at hundreds or thousands of companies. But is it that having a why leads to success? Or is it once you're successful, you have the time and headspace to come up with a purpose statement because you're not firefighting? Or is it A third factor causes both. A great CEO causes good performance. A great CEO comes up with a purpose statement, but there's no direct link between the two. And so why do I call this misstep? Data is not evidence. It may not be conclusive, because what is the difference between data and evidence? We often hear these terms interchangeably. But what is evidence? Now, we hear evidence in the context of a criminal trial. Evidence is evidence that points to one particular suspect. If the evidence suggests that Tom or Dick or Harry could have killed Sarah, that is not evidence. It's consistent with multiple explanations. So what we would want to move from data to evidence is not just a correlation which has multiple interpretations, but something which may be causation, a way of ruling out is it success that leads to a why, or is it good management that leads to both. But again, if we have confirmation bias with a side of a nice narrative, then then we latch onto the causal exhalation, which is the why leads to success.
- [46:29]
B
How big of a problem is data mining in research? Generally speaking, it's a huge problem.
- [46:34]
A
And so data mining, this is a different issue from correlation versus causation. This is the problem that there might not even be a correlation to begin with. Why? Because you fished for it, you've mined for it. And so what do I mean by this? Again, let me give an example. So let's say I want to link sustainability and financial performance. Well, there's so many ways I could measure sustainability. Maybe I could look at carbon footprint. I could look at water usage. I could look at biodiversity impact. I could look at how you treat your employees. I could look at gender diversity, racial diversity. The list is endless. And in terms of financial performance, I could look at sales growth. I could look at profitability. I could look at total shareholder return. I could do this over five years or 10 years or one year. Even if I came up with a robust correlation that sustainable companies perform better, you need to ask yourself, well, could there have been many, many other ways of measuring sustainability and performance that the researcher did, and they only report the one that gave them the findings that you want? But how does a reader discern this? Because you have no idea about all the other tests that they ran. But it's to ask yourself, do you have the most natural measure of performance and sustainability? For example, McKinsey has a ton of reports which link sustainability or other things like diversity to ebitda, which is earnings before interest, tax, depreciation, and amortization. So that is one measure of performance. But those common measure of performance if we just look at how the stock price does. And so why is that much better? Because profitability only measures the short term, yet the stock price is forward looking. Some of the best performing companies nowadays are tech firms which don't have high current profits. But because they're expected to have higher future profits, the stock price is already encapsulating that.
- [48:37]
B
Can you share the example of an investor that had come to you wanting to build a fund around some performance measure and you did the research and it wasn't there, but they went and did the fund anyway?
- [48:46]
A
Yes. So this is one of the world's leading investors, who I give a pseudonym to in the book. But she had heard of my research linking employee satisfaction to long term firm performance and she wanted to launch a fund based on diversity. I was really excited. I thought, okay, I could work with this leading investor. I myself am an ethnic minority, so I would like to believe that diversity pays off. Off. And there were 24 measures of gender diversity, which was her specific lens, which I tried to correlate with performance. And I did. And I found in 22 of those 24 cases there was sadly, a negative link. Now it was clear what I should do if I wanted to work with this investor, I could just disclose the two positive ones, but that would not be correct in terms of honesty. So I disclosed all 24, and she wasn't interested in working with me. She politely thanked me, but then moved on. I thought, well, this is going to be the end of it. And then just six months later, I saw her launch the fund with a big fanfare that this is backed up by scientific research showing a strong link. And I was surprised because I had done the research 24 times and not found it. And what she had found was research by another company linking diversity not to shareholder returns, which is what any investor should care about, but measures like the return on sales or return on capital employed that had a very specific measure of diversity. It compared boards with three or more women to boards with zero women. But why do you set three or more versus zero? Why is it not three or more versus two or fewer, two or more versus one or fewer? Or why isn't it something percentage wise, at least one third women and two thirds men or something? It may well be that this 3 versus 0 was the only thing which worked. And so that's why they reported this. And so this matters because this fund and many other funds with the same theme, they have all underperformed. And so if you're basing this on misleading data, it doesn't matter how much of an advocate you are for the cause, and it may well be seen to be a worthy cause. But do not back up this cause with some incorrect data, just as as a police officer, you might have a high conviction that this person is guilty. And it may well be that the court has just unfairly made certain rulings that certain evidence is inadmissible. But despite how strongly you feel that is not just. It is not an excuse to then make up data or make up evidence to convict your suspect.
- [51:21]
B
Seems like a really tricky thing in financial economics. As a practitioner, people used to say evidence based investing was a good thing to imply things like index funds. I found that phrase to become increasingly meaningless because you can find evidence or a form of evidence at least to back up pretty much any investment strategy that you want to make a case for.
- [51:39]
A
We like to throw around these phrases like research finds that study proves that, but you can find a study to show whatever you want to support. What matters is the quality of the research. And also not just to be skewed by one study, but, but to look at scientific consensus, which is the findings of multiple studies.
- [51:57]
B
What can people do to guard against data?
- [52:00]
A
Money, as a consumer of research is to ask, well, have you looked at the most plausible ways of measuring an input and an output? So let's say the output is performance. The most natural measure for any fund is total shareholder return. That's the measure of performance. They always advertise. So if you look at return on sales, that is quite odd. And also the input, if you're looking at three or more versus zero, that's strange because what about the board with one or two women? Why are we looking at the absolute number of women on the board? Shouldn't it be a percentage? Because if a board is larger, you might expect more women than if the board was small. So just to ask yourself, are there more plausible ways of measuring this? And if there were and they're not reported, then that might raise a red flag. But what the best studies will do, we'll do a robustness check, which is to show, well, if we were to measure performance in this other different way, we still get the same results. If we measure diversity in this different way, we looked at percentages, non absolutes, we still find this.
- [53:01]
B
So moving up the ladder, when is data evidence?
- [53:06]
A
So that is when you have something like a randomized control trial. So in medicine, I have randomly assigned some people to get the drug and some people to get a placebo. Now in real life, outside of randomized trials, you might think, well, that's pretty rare. But sometimes you do get the equivalent, which is known as a natural experiment. So what do I mean by a natural experiment? Well, a lab experiment is what I have just described. Placebo and a treatment. A natural experiment is when this separation applies naturally, perhaps because of a law change. There was a time where I hope I'm getting the states the right way around, where Pennsylvania passed a minimum wage, but New Jersey, which is just across the border, did not pass a minimum wage. And so it's like the New Jersey restaurants had were a placebo. Nothing happened. The Pennsylvania restaurants were the ones with the treatment, with the drug. And so you can compare the effects of the minimum wage, unemployment. And the study I cite in the book finds surprisingly, the minimum wage of anything increased employment compared to the common perception that minimum wages are always damaging to employment. Why you compare two very similar restaurants with similar economic conditions? Because they're just very close to each other. Just one is in Pennsylvania, the other is in New Jersey, and then one was affected and the other was not.
- [54:31]
B
What does this say about evidence in financial economics where natural experiments and RCTs. Well, RCTs I don't think you can do. But natural experiments are pretty rare as well.
- [54:41]
A
It's actually quite difficult to get evidence which is conclusively causal. And does this mean that, well, we can't believe anything and anything goes? No, but it's just to say that we don't want to rely on things too heavily. So absolutely the gold standard should not be causal, 100% evidence. Otherwise we could just be ignoring many other things which might be 80% true or 70% reliable. But instead it's not to view things as black and white. It's not either completely nailed and completely invalid. Even if what you find is a correlation, that could still be useful. And indeed, some of my own papers only find correlations because I don't have a natural experiment. So one example is my work on cognitive diversity. Now, we do have natural experiments on demographic diversity, which is forcing say, boards to have 40% of women. There is no natural experiment forcing there to be a diversity of different viewpoints. That's much harder to pass a law about. So all I have is correlations. But those correlations at least can be suggestive. They might get us 70% of the way there, even if it's not completely conclusive.
- [55:50]
B
When we have evidence, why is evidence not proof?
- [55:54]
A
So let's say we've nailed causality. We've got a perfect natural experiment that might only be true in that particular setting, it may not be universal. So what is the difference between evidence and proof? A proof is universal. So when Archimedes proves the area of a circle that was true not only in ancient Greece in the third century BC when he did his proof, it's true in circles in 2025 in Canada today. But often when we look at evidence, it might be gathered only in one particular setting. Let me give you an example. So there was a very famous book by Angela Duckworth and a really famous TED talk about the power of grit, the power of passion and perseverance. And what she did to show that grit mattered was she took men and women who got to West Point, which is the United States Military Academy. If you get into West Point, you're not yet in the army, you have to survive a six week course called the Beast barracks, which is really tough, it's mentally and physically demanding. And she wanted to see, well, what predicts survival of beast barracks. You might think it's physical fitness. But she found the one thing even more important than fitness was grit, which she defines as passion and perseverance. So that is striking. So grit must be so powerful if it's even more important than fitness in a physical challenge. But the formula here is something known as restriction of range. So what is that? She had a sample of already very fit people. So in order to get into West Point you already have to be extremely fit and maybe they had already passed the threshold level of fitness and that's why fitness didn't matter, because everybody was already fit enough to survive Beast barracks. Now it's something else, like Rick Matlab. But we cannot over extrapolate from that setting of really fit people and to say, well, for the general person on the street, work on your fitness, work on your grit, not your fitness. No, for the ordinary person dreaming of joining the military, maybe he or she should work on their fitness. That's the more important thing.
- [58:02]
B
Yeah, that's interesting. So the grit research, it's not wrong, but it's right only within the specific setting in which it was conducted?
- [58:10]
A
Yes, and that's true for many other studies. So a lot of studies are done on weird people. You might think, well, that's strange for me to say, call somebody weird, but weird, Central, Western, educated, industrialized, rich and democratic. That's where a lot of studies take place and some of them we can generalise. So if it's something like smoking causes cancer in Canadians, that is something where it might well apply the same to Latin Americans or Chinese people, because that's something which is to do with the body. But if it's something cultural, which is, say, the link between equality of pay and perceived fairness, there may well be different social norms. So you're never going to get a study on the particular setting that you're interested in. But again, perfect should not be the enemy of the good. We want to ask ourselves, based on logic, is this a setting which is sufficiently similar and are there logical reasons to believe that it won't apply here? Or with smoking and cancer, I don't think we really need to be so specific in terms of context, but for something more cultural, behavioural, then the context may matter more.
- [59:14]
B
We've established that the truth is hard to establish. What steps do you think people should take to seek the truth when they're making an important decision? You've mentioned busy people a few times throughout the conversation. I guess within that context, what should people be doing?
- [59:28]
A
So it's to be discerning about any evidence. And so what I have in the back of the book is just a checklist of simple questions to ask yourself, following the ladder inference. So if there is a statement, is there evidence behind this or are we just misquoting? If it's a fact, is this one anecdotal? Is this here in large scale? If it's large scale data, is there alternative explanations? So it might not be a causation? And even if there is causation, what was the setting in which it was gathered? Is it different to what setting we care about? But more generally than just evaluating studies, because studies are not the only source of information, we get information from other people's opinions is just to go out there and actively seek different viewpoints and make it clear that you are open to different viewpoints, particularly if you are a leader or in an authoritative position like a professor, where people might incorrectly assume that they should not challenge you. And when there is challenge, it may well be that you want to make it clear how much you appreciate it. So let's say you're the chair of a board and then you're going to go ahead with a strategy and then somebody raises an objection, you hear that objection and you discuss it open mindedly. But people think that even though that objection is well founded, the benefits outweigh the costs and you still go ahead. Maybe you should go to that person afterwards, or maybe you should do this publicly to the rest of the board and say, I really appreciate you sharing your different viewpoint. Even though we still went ahead with the decision. We will bear your concerns in mind when we execute the strategy. And in the future, if you have concerns, please continue to voice them. If you had not done that, what might the dissenter think? He or she might think, well, I had to sacrifice my political capital to raise this concern. I may have annoyed the people who proposed the strategy. I delayed the meeting by half an hour and I didn't change the decision anyway. So in the future I'm just going to self censor. So if you make it clear that you do encourage different viewpoints, then this does mean that you're going to still get differences of opinion and you're going to get closer to the truth.
- [61:34]
B
What about at a societal level? What should we be teaching people and thinking about to embed these ideas that you're talking about in our everyday lives?
- [61:42]
A
It's to teach the value of critical thinking. You might think, well, in a societal level, is this feasible? Can we teach this in schools? And I think we can. Even though, yes, statistics we might only learn when we're 16. But what I'm highlighting is statistics is not about just numbers. There is not a single equation. Lifebook. What statistics is about is looking at alternative explanations is about being discerning. So just like we teach a kid, don't accept sweets from strangers, there's an alternative explanation for why the stranger is giving you sweets. It's a nefarious one. We also teach kids well, not directly, but when they read murder mysteries, they learn quickly that it's not the most obvious suspect who committed the murder. They look for alternative suspects. Given that kids are able to establish and grasp that, I don't think it's unrealistic to teach them the power of thinking about alternative escalations. And this is valuable not just for the interpretation of data and statistics, but more civil discourse. What we see right now is sometimes people canceling those with different viewpoints. We see echo chambers. We see a lot of mental health issues when a student might be cancelled by the rest of his or her peers. But if we highlight that even on issues that we think are pretty black and white, there may well be two sides to it, then we are able to encourage people to have a greater respect for opinions that might differ from ourselves.
- [63:09]
B
Man, accepting candy from strangers as an analogy to reading data that confirms your prior beliefs is pretty good.
- [63:16]
A
Yeah, and it's actually not too far fetched because this is indeed why things are so appealing. So just like candy is appealing, we want to accept it. We don't want to ask the question. We simply would like to believe that study and to consume it just like you want to consume the candy.
- [63:31]
B
Yeah, awesome. All right, Alex, that's all my questions. We really appreciate you coming back on the podcast. This has been great.
- [63:36]
A
Thanks so much for having back. Really enjoyed the conversation.
- [63:43]
C
Portfolio management and brokerage services in Canada are offered exclusively by PWL Capital, Inc. Which is regulated by the Canadian Investment Regulatory Organization and is a member of the Canadian Investor Protection Fund. Investment advisory services in the United States of America are offered exclusively by OneDigital Investment Advisors, LLC. OneDigital and PWL Capital are affiliated entities. However, each company has financial responsibility for only its own products and services. Nothing herein constitutes an offer or solicitation to buy or sell any security. This communication is distributed for informational purposes only. The information contained herein has been derived from sources believed to be accurate, but no guarantee as to its accuracy or completeness can be made. Furthermore, nothing herein should be construed as investment tax or legal advice and or used to make any investment decisions. Different types of investments and investment strategies have varying degrees of risk and are not suitable for all investors. You should consult with a professional advisor to see how the information contained herein may apply to your individual circumstances. All market indices discussed are unmanaged, do not incur management fees, and cannot be invested indirectly. All investing involves risk of loss and nothing herein should be construed as a guarantee of any specific outcome or profit. Past performance is not indicative of or a guarantee of future results. All statements and opinions presented herein are those of the individual hosts and or guests, are current only as of this communication's original publication date, and are subject to change without notice. Neither one Digital nor PWL Capital has any obligation to provide revised statements and or opinions in the event of changed circumstances.