← You Are Not So Smart
You Are Not So Smart
337 - Cognitive Surrender - Gideon Nave and Steven D. Shaw
00:59:39
How is AI reshaping human reasoning? What is cognitive surrender, and how do we avoid its negative impact? What is system three thinking, and how can we get the most out of it?
Loading summary
Transcript64 lines
- [00:01]
David McRaney
America's best network just got bigger.
- [00:04]
Dr. Stephen D. Shaw
Switch to T Mobile today and get
- [00:06]
David McRaney
built in benefits the other guys leave out. Plus our five year price guarantee. And now T Mobile is available in US Cellular stores. Best mobile network based on analysis by Oogle of speed test intelligence data 2H 2025. Bigger network. The combination of T Mobile's and US cellular network footprints will enhance the T mobile network's coverage price guarantee on talk text and data exclusions like taxes and fees apply. See t mobile.com for details. You can go to kittedkitted shop and use the code Smart50Smart50 at checkout and you will get half off a set of thinking superpowers in a box. If you want to know more about what I'm talking about, check it out. Middle of the show, Cooling of the happy.
- [01:03]
Dr. Stephen D. Shaw
They went quiet. They went fire.
- [01:13]
David McRaney
Welcome to the you are not so smart podcast. Episode 337. Proceed Straight. There's a scene in the television series the Office, the US Version of the Office, in which two characters, Michael, the boss, and Dwight, one of his employees, are in a car on a work trip using gps, something akin to Google Maps or Apple Maps. And Michael is driving, Dwight is in the passenger seat. And the GPS tells them they should turn right even though the road that goes to the right is visible ahead of them. And if they were to turn to the right where they are at that moment in their drive, they would crash through a fence and go straight into a lake.
- [02:19]
Dr. Gideon Nave
And White basically tells Michael, Michael to override the gps. And Michael tells him, no, the machine is right. The machine knows.
- [02:29]
David McRaney
307.
- [02:30]
Dr. Gideon Nave
Make a right.
- [02:31]
David McRaney
Maybe it's a shortcut. Dwight, it said go to the ramp. It can't mean that there's a leg there.
- [02:35]
Dr. Stephen D. Shaw
Sink.
- [02:35]
David McRaney
It knows where it is going. The machine knows. Stop yelling at me.
- [02:39]
Dr. Stephen D. Shaw
No, it's stop yelling. There's no room here.
- [02:43]
Dr. Gideon Nave
Dwight was doing what we would say is offloading, right? He's using the GPS as a tool to do something that he cannot do as easily and as quickly. Michael, on the other hand, is completely giving up his critical thinking and just following blindly what the GPS tells him to do. And this is leading to this catastrophical error.
- [03:06]
Dr. Stephen D. Shaw
Remain calm.
- [03:07]
David McRaney
I have trained for this. Okay, exit the window now.
- [03:11]
Dr. Gideon Nave
Most of us, when we use gps, we are offloading. We are not surrendering. And in some rare cases, I mean, the story of the Office is to some degree a reflection of maybe something that happens in our reality when we really have no idea where we're going. We sometimes can Trust the machine more. But to this degree, when we are not even using our critical thinking, when we are about to drive into a lake, that's like a complete surrender. So I would just put forward this distinction between these two different processes.
- [03:44]
David McRaney
Here we go.
- [03:45]
Dr. Stephen D. Shaw
Make a U turn. If possible.
- [03:47]
David McRaney
Look out for lasers. The person who was just Speaking that was Dr. Gideon Nave, a doctor of computation and neural systems who studied at Caltech and is now a professor and researcher at the Wharton School of Management. And what he is describing here is an example of something he and his colleague, Dr. Stephen D. Shaw, who you will meet shortly, are calling Cognitive surrender. And this office episode example is the example that Dr. Navi gave me when he wanted to quickly and simply define that term, cognitive surrender. In contrast to another term used by the sciences that study the brain and the mind called cognitive offloading, when you use a calculator, a spreadsheet, even a notebook or a to do list, that's cognitive offloading. It's the act of using something external, or even using some kind of action or behavior to aid in your cognition. You alter the requirements needed to process information to reduce the cognitive demand of a particular task. That's cognitive offloading. Cognitive surrender is something else. Here we go.
- [05:12]
Dr. Stephen D. Shaw
Make a U turn if possible.
- [05:17]
David McRaney
My name is David McCraney. This is the youe Are not so Smart podcast. And in this episode we are exploring something that each of us will need to become familiar with if we are going to add AI, LLMs, Digital Assistants, and other similar forms of technology to our lives and to our workflows, or if we choose to not add them to our lives and our work and just intend to navigate a world in which other people are using those things. We're still going to need to become familiar with what we're going to talk about today. Either way or some way in between our guests in this episode, two scientists who recently released a study into all of this suggest that your critical thinking toolkit will need to be updated to account for some habits in our cognition that large language models, also known as generative AIs, stuff like ChatGPT, Gemini, Claude, and so on, tend to trigger habits that can produce serious negative outcomes. That's what they found when they did research into this, and that research was just published under the title Thinking Fast, Slow and Artificial How AI Is Reshaping Human Reasoning and the Rise of Cognitive Surrender. Let's meet the authors of that paper. Here is Dr. Gideon Nave.
- [06:45]
Dr. Gideon Nave
Hey. So I'm a professor of marketing at Porto. I have quite unusual trajectory at Academia. I started as an electrical engineer, actually learning to train machine learning models and all sorts of other things of this sort when it was a very small niche in the engineering school. And then I figured out that I'm actually more interested in the human brain than in developing computers that are capable of imitating it. I always tell my students that we're kind of humbled by AI now, but the brain does many things better than AI with far less energy and with far less training sample. And I think we still have a lot to learn from the brain, even if something that is consuming an entire power plant a day is doing better than it.
- [07:44]
David McRaney
And this is Dr. Stephen D. Shaw.
- [07:48]
Dr. Stephen D. Shaw
I'm a postdoctoral researcher at the Wharton School right now. And my academic undergrad was more in the biological sciences. I studied microbiology and then genetics and neuroscience. For a while I was doing research on the neuroscience of music and beat perception. And then I went to the to graduate school for a PhD at the University of Michigan. Go blue. And you know, now I'm here working on cognitive surrender to AI.
- [08:18]
David McRaney
Yeah, yeah. So Nave and Shah, these are two scientists who are deeply fascinated with cognition, with minds, with brains, with artificial intelligence, the interplay between brains and what we are currently calling artificial intelligence. And they were deeply concerned with how readily people seem to trust the current models. ChatGPT, Claude Grok, Gemini, maybe even Microsoft Copilot. And the key word here is trust. This is what they were discussing back and forth in their academic lounge areas and labs and such. Trust. Of these models, trust is a very human mix of cognition, emotion, awareness, conscious subjective processing of perceptual modalities and that sort of thing. It's trust. It's a flesh and blood human estimation. It's a feature of human to human relationships. And yet people often find themselves trusting what we are currently calling AIs. And I keep referring to them that way because they aren't quite at the level of our sci fi concepts of artificial intelligence. LLMs are still very far away from Lt. Data of Star Trek fame or the replicants in Blade Runner, but they are very good at emulating what we might imagine would be the output of those sorts of artificial intelligences. Even though it's just words on a screen, they're generating a sense of trust, a sense of agency. And we engage with these cues through very ancient mechanisms of social interaction. And we can treat them as if they are friendly and authoritative, which has an impact on our cognition. And more and more people are more and more often using modern day AI when working on tasks that involve reasoning, not just sorting lists or planning out things. They're using it to engage in critical thinking and problem solving, to facilitate decision making. And we're beginning to use these outside of a professional setting, using them to optimize child care, to get relationship advice, to work through financial issues, to make workplace decisions outside of the workplace. So, and we're going to deep dive into all of this in a moment. Naveh and Shaw conducted a study into what happens when people use AIs in that way to go beyond a super advanced search engine and use it to aid their intuition and to summarize what they discovered. Even when the AI was very, very wrong, once people had asked it for help and it had given what seemed like a helpful, authoritative answer, people tended to engage in cognitive surrender. They drove into the lake. In other words, they went with the answer and didn't even consider if it was right or wrong. And what makes cognitive surrender so strange is that it involves two errors, two thinking mistakes. Here is Stephen Shaw's description of those two mistakes.
- [11:52]
Dr. Stephen D. Shaw
One, adoption of AI outputs without verification and two, treating AI's outputs as one's own.
- [12:01]
David McRaney
During a back and forth with an AI, Nave and Shaw found that not only were research participants prone to setting aside their critical thinking, their skepticism, their fact checking instincts, and in so doing, readily believing its output was accurate even when it was not. Somewhere in that exchange of prompts and responses we can lose track of where the information is coming from because in the study the participants took ownership of the AI generated answers. Which suggests that when AIs produce insights or conclusions or advice, we may sometimes treat it as if it comes from within. As if the AIs simulated aha moments are our aha moments. So how is this happening? Why is this happening and what should we take away from all of that? Well, we're going to get into it all of that after this. Commercial break. America's best network just got bigger.
- [13:20]
Dr. Stephen D. Shaw
Switch to T Mobile today and get
- [13:22]
David McRaney
built in benefits the other guys leave out. Plus our 5 year price guarantee and now T Mobile is available in U S Cellular stores. Best Mobile Network based on analysis by Google of speed test intelligence data 2H2025 bigger network the combination of T Mobile's and US cellular network footprints will enhance the T Mobile network's coverage price guarantee on talk text and data exclusions like taxes and fees apply. See t mobile.com for details.
- [13:46]
Dr. Stephen D. Shaw
So good, so good, so good.
- [13:49]
David McRaney
New markdowns, up to 70% off are
- [13:51]
Dr. Stephen D. Shaw
at Nordstrom Rack stores now. And that means so many new reasons to rack because I always find something amazing. Just so many good brands because there's always something new.
- [14:02]
David McRaney
Join the NordicLub to unlock exclusive discounts. Shop new arrivals first and more. Plus buy online and pick up at
- [14:10]
Dr. Stephen D. Shaw
your favorite rack store for free. Great brands, great prices.
- [14:14]
David McRaney
That's why you rack the School of Thought I love this place. I've been a fan of the School of Thought for years. It's a non profit organization. They provide free Creative Commons critical thinking resources to more than 30 million people worldwide and their mission is to help popularize critical thinking, reason, media literacy, scientific literacy, and a desire to understand things deeply via intellectual humility. So you can see why I would totally be into something like this. The founders of the School of Thought have just launched something new called Kitted Thinking Tools K I T T E D Thinking Tools and the way this works is you go to the website, you pick out the kit that you want, there's tons of them and the School of Thought will send you a kit of very nice, beautifully designed, well curated, high quality. Each one about double the size of a playing card, matte cello 400 GSM stock prompt cards and a nice magnetically latching box that you can use to facilitate workshops, level up brainstorming and creative thinking sessions, optimize user and customer experience and design, elevate strategic planning and decision making, mitigate risks and liabilities, and much, much more. Each kit can, if you want to use it this way, interact with this crazy cool app. Each card has a corresponding digital version with examples and templates and videos and step by step instructions and more. You even get PowerPoint and Keynote templates. There's so many ways you could use this. Here's some ideas. If you're a venture capital investor, you could get the Investors Critical Thinking Kit and use it to stress test and evaluate different startups for Series A funding. If you're a User Experience Designer, you can get the User Design Kit to put together a workshop with internal stockholders for a software product. Or if you're an HR professional, you could mix and match these kits to create a complete professional development learning program tailored specifically for your team over the course of the next two years. So if you're the kind of person who is fascinated with critical thinking and motivated reasoning and intellectual humility and biases, fallacies and heuristics. You know the sort of person who listens to podcasts like you are not so smart. You're probably the kind of person who would love these decks. If you're curious, you can get a special 50% off offer. That's right. Half off offer right here. You can get half off off of one of these kits by heading to Kitted Shop K I T T E D Shop, and using the code smart50 at checkout. That's smart50 at checkout. 5% of the profits will go back to the school of thought. So you're supporting a good cause that distributes free critical thinking tools all over the world on top of receiving a set of thinking superpowers in a box. Check all of this out at Kitted Shop or just click the link in the show notes. And now we return to our program. I'm David McRaney. This is the youe Are not so Smart podcast, and we are exploring something in this episode called Cognitive Surrender. And before we return to Dr. Navi and Dr. Shah, I wanted to add, as some foundation, that unless you add a prompt to your AI asking it to stop telling you that you are very smart, that you're a very smart person and you have very good taste, and your current goals are very good goals, many current models will, by default, tell you those kinds of things. And I say tell you, but it's just letters appearing on a screen coming out of a program. But it seems like it's telling you things, and they tell you those kinds of things because someone within the company that is offering you that product has decided it should do that. And even though you very likely understand that it can't actually feel that way about you and what you are telling it because it can't actually feel feelings, those words can still be quite powerful. They can affect not just your interactions with the AI, but your cognition, your emotions, your behavior, even when you're away from the AI after the interactions. And that can happen because, in a sense, the AI is engaging in a form of mimicry, social mimicry. And that mimicry works on us because certain social cues trigger ancient heuristical responses. I mean, words are powerful. A poem can bring you to tears. A movie can make you feel connected to fictional characters, even when you're well aware that those are actors and there are cameras and all of it is make believe. These triggers, they remind me of pet beavers. Yes, pet beavers. There are people who have rescued beavers, very kind people who are involved in wildlife rescue. And they will sometimes end up with a beaver in their homes that has never known anything but human interaction and human environments. Yet when those beavers hear the trickle of water coming from a faucet, they will go around the house and collect towels, pillows, potted plants, and start building a dam, like in a hallway or something. A beaver is a fairly complex mammal with bones and eyes and ears and stuff. And building a dam is a fairly complex set of behaviors. When they do that, they are detecting a pattern that a portion of their brain built by their DNA responds to. Those instructions are the result of millions of years of evolution. Things in the environment over millions of years that affect the DNA that built their brains and thus produces that behavior, that compulsion in the presence of that pattern. If the perceptions are getting that pattern and delivering it to their brains, that behavior is the result. In psychology, we'd say that is an environmental cue that triggers an innate behavior. In the 1930s, an ornithologist, a bird scientist named Nicholas Tinbergen, noticed that herring gull chicks would peck at a red spot on their parents beaks to get their parents to feed them both sides of that behavior, that dynamic, that call and response, all innate. So he did some studies in which he presented chicks of that species with a fake beak with a much larger and more red redder spot on it. And when you would present those chicks with those bigger beaks with bigger redder spots, they would peck at those spots with even more intensity and regularity than they would real beaks with real red spots. So, like the beaver, it was an environmental cue that triggered an innate behavioral response. But he showed that you can enhance one of those triggers to be what we now call supernormal releasers. Cues more powerful than what a species would find in nature. And the resulting behavior is more exaggerated than what you would find in nature. And we have plenty of studies like this. Mother birds will leave their real eggs for bigger fake eggs. Insects will mate with replica insects that have the same shapes and patterns as what they expect, but more. These are species specific, instinctive chains of behavior that once triggered by a close enough stimulus or greater than expected stimulus, will run in sequence to completion without any further stimulus required. And yeah, we have stuff like this in our brains too. There's a portion of our brain that is devoted to recognizing faces, even at the expense of false positives. That's why you see faces in electrical outlets and grills of cars, the moon, a piece of burnt toast, when we see faces and stuff like that. It's called pareidolia. And I mention all of this because we are rather susceptible to something that I'm going to call agentic pareidolia. There was this toy introduced in the late 1990s called the Furby. And furbies Were these animatronic owls like owl toys that played pre recorded gibberish sounds? Over time though, they played less and less gibberish and more and more pre recorded English. They did all sorts of other stuff too, in response to light and dark and being jostled about. So people who were playing with them felt like they were interacting with them, right? And over time, since they were talking to the Furby and it started to produce less gibberish and more English, they believed it was picking up on what you were saying, that it was learning, and that was all by design. The creators of the Furby wanted people to think that, and many people believed these toys had agency, that they were alive, that they were learning, they had minds. Even though all of that stuff was pre planned, pre recorded, even if you accepted they were toys, many people believed those toys had robot minds. If they accepted that they were just animatronics, they believed they were AIs of some kind. So the thing is, if you give us just the right amount of the right kind of life form cues, we will fill in the rest with agentic pareidolia. Because there's such a rich body of psychological literature into this sort of thing, and because we've already done so much research into the impact of advertising and television, movies, newspapers, social media, algorithm fueled short form videos, psychology is pretty sure AI is another technology that will impact our behaviors in some way. It's just that thanks to all this agentic pareidolia happening in a problem solving environment, a decision making dynamic, the impact may affect our critical thinking and our judgment. We just don't know exactly how because research takes time. And AI has been around just long enough so that the first wave of psychological research into human AI interaction is just now starting to be published. Which brings US Back to Dr. Gideon Nava and Dr. Steve Shaw, two scientists who conducted one of that wave's first major studies. And they based their research on a solid foundation, something in psychology known as dual process theory. You've probably heard of this. It's often called thinking fast and slow. Thanks to legendary psychologist Daniel Kahneman, who wrote a massively best selling book about his research into that very topic with that very title. Here is dual process theory. Dual process model in a nutshell. When it comes to solving problems, reaching goals, making decisions, humans use two cognitive systems. System one is mostly heuristics and innate responses. If we can get away with being biased and lazy, we tend to sacrifice accuracy for speed. In other words, we go with our guts. Especially if the answer just Seems obvious. But if we have the time and energy and feel significantly motivated, or our attention becomes alert to an error in our predictive processing, System two will come online. We will engage in more careful problem solving. You will enter a more contemplative state of mind. And as research has shown, oftentimes the answer that seems obvious in System 1 thinking, once System 2 thinking is applied to it, it turns out the obvious answer was incorrect. While Navae and Shaw were contemplating the impact of AIs on human reasoning and cognition, Navae proposed that we may need to advance dual process theory into a trisystem theory.
- [26:36]
Dr. Stephen D. Shaw
Giddy is my supervisor, my boss in many ways, right at Wharton. And so we always just bounce around ideas, trying to come up with what we're interested in. And one day he said, thinking fast, slow and artificial in a meeting. And I was like, well, that's pretty cool. And he knew it was cool too. And I started thinking a lot about AI as a thinking system, as part of human cognition as well. Human AI interaction. Right. And how does it change and reshape the current systems, the dual process, system 1 and system 2? Intuitions and deliberations where you can allow, or we can allow, and we see it in ourselves or we see it in others. You can allow AI to sort of think for us.
- [27:21]
Dr. Gideon Nave
I think, you know, the naive way is going to say, okay, there is system one and system two, there is intuition and deliberation. Let's just put plug in another system and that's it. Theory is done. But that's not what we are into. What we are into is how the introduction of the system that does outsourcing, outsourcing of our thought, how does it influence the two other systems? How does it influence our cognition? So there was a lot of back and forth on that. And then the second part is the experiment. And devising an experiment is not that easy.
- [27:57]
Dr. Stephen D. Shaw
Yeah, and so, you know, we're thinking about moving from dual process from, you know, the world before AI to add system three. Adding AI. And so we, you know, thought, well, how. What's the best way to do that experimentally? Wouldn't it be cool if we used a task that's been used for many other dual process papers and then basically show that when we add AI, it changes performance, accuracy, confidence, all sorts of things on that task? So that task is called the cognitive reflections task or the crt. And it's basically a set of seven logic and reasoning problems, some simple math problems or logic games like this. And when you read the Problem, there's an answer that comes to mind almost immediately. This is the intuitive answer to the system one answer. But you can sort of check that answer pretty easily. And if that, you know, if you do a little bit of verification, you'll see that that intuitive answer is not correct. And so you have to do a bit of deliberation to be able to find out, okay, I've got the correct answer. You can do some math, or you can do trial of error, whatever you'd like.
- [29:06]
David McRaney
So here's a question from the cognitive reflection task, so you can get an idea of what's going on here. If it takes five machines, five minutes to make five widgets, how long would it take 100 machines to make 100 widgets? Now, notice your immediate intuitive answer. And now take however long you like, pause the show if you need to, to work out what is the actual accurate answer to that question. And I'm going to assume you've done that, because I'm going to return to the interview now.
- [29:44]
Dr. Stephen D. Shaw
And so we have basically, our control condition is the old world, the dual process world. We just have that task, it's very well calibrated, and people get the answers right about 50% of the time. And then in the experimental condition, we say, we have the same questions, seven questions, logic and reasoning, and we say, okay, what you have access to ChatGPT on the side here, a terminal, and you can use it if you want to, but you don't have to use it. We tried to make it naturalistic. He said, you don't have to use it, just go up. But if you want to, you can and use it however you want to. And so we saw people using it quite a lot. Over 50% of trials, they chose to use ChatGPT. But the trick that was the really clever trick on the back end was that, and the participants don't know that, is that we manipulated the answers of ChatGPT behind, behind the scenes. So if they consulted ChatGPT in that window about the one question that was on the screen, we experimentally manipulated whether it gave them the correct answer or the incorrect answer. And that's how we were able to causally see cognitive surrender if they asked it about anything else. And this is also key to being naturalistic, they could talk about it about Winston Churchill or about Pokemon cards, or about Furbies, you know, if they wanted to, but it would answer just like ChatGPT would. But if they ask it about that one very specific question, right, which we hypothesize is what they would do right. They would use it and they would basically copy and paste it in there and get the information and then adopt it. It would give 50% of the time the right answer, 50% of the time the wrong answer. And for each of those incorrect or correct answers, it would give some set of reasoning behind it, some support, some basic, this is why this is a correct answer. And even that support we didn't constrain, we just made sure that it was supporting the correct or incorrect answer. But we let ChatGPT do the reasoning behind that. And basically what we end up seeing is that people adopt those answers once they've gone to the chat. Once they've deferred and asked the question, they adopt those answers at quite high rates. And so we see accuracy increase quite a lot when AI is giving them correct information and decrease quite a lot when it's giving them incorrect information. And that is cognitive surrender right there.
- [31:59]
Dr. Gideon Nave
Yeah, I would say the key pattern of cognitive surrender is that it goes below baseline, so that there is a chance here that if the AI is wrong, you're going to be actually worse off, as opposed to just using your own brain.
- [32:15]
David McRaney
The cognitive reflection task has been around for a long time, and it is very reliable, at least in producing all sorts of interesting effects. And it's very simple. You get this question that seems like it has an obvious answer, but the obvious answer is incorrect. So if you just go with your gut, you go with your intuition, you tend to get the wrong answer. But if you engage in cognitive reflection, if you engage in system two thinking and just kind of think about it for a second, work it out, you'll get the right answer, you'll notice. Oh, okay. It's actually a little more complicated than it seems. And it's not testing whether or not you get the right answer. It's testing will you engage in cognitive reflection and what affects that? It turns out there's all sorts of things that can affect whether or not a person will engage in cognitive reflection. It's not necessarily a test of a person's intelligence or cognitive ability. It's a test of when will we engage in system 2 thinking and when will we not? And it could be just as simple as not getting enough sleep or you're very stressed, or there's all sorts of things happening around you that are distracting. Many things can affect whether or not you will engage in system 2 thinking and thus go with a more contemplative state of mind. To answer this question with a little bit more deliberation than you would otherwise. In this experiment, they gave people these questions, but they gave them the option to, instead of engaging in deliberation, use ChatGPT. And if they did choose that, which many people did, ChatGPT would, half of the time give people the wrong answer. But whether or not it gave them the right answer or the wrong answer, people tended to engage in cognitive surrender. They went with that instead of doing their own thinking. They bypassed system two and let chat GPT do that. And as Dr. Gideon just said, if the answer is wrong, you'd have been better off in this study using your own brain. What's the difference between using this and a calculator? What's the difference between using this and like, as you were describing, like, Google Maps or something like that? Or by extension, what if it's a tool? Like, what's the difference between using this and a spreadsheet or a word processor or Autocorrect? And then even further out, like, are you a Luddite? Are you saying we should stop using steam engines and printing presses or whatever? Right. Just for that world of pushback? What are we talking about here when we say cognitive surrender versus being a cognitive miser, cognitive offloading and all the rest? What makes that sort of a unique set of behavioral routines and cognitive doodads going on?
- [34:57]
Dr. Stephen D. Shaw
Yeah, so like a calculator or search engine. The way we frame these in the paper is that they allow us to cognitively offload. So you can take certain components of cognitive capabilities of thought, whether it's calculus or memory or different sort of components, and offload them. The argument that we're making here is that System 3 AI is sort of a fundamentally different technology. And the reason why we make that argument is because, for example, you talked a lot about trust, and you might trust a doctor with information about your health or a lawyer with information about legal stuff. They're going to have specific expertise about those domains, but you're not going to have access to them all the time. And they're not going to be experts on everything that's out there. Even if you have maybe a family member who's a lawyer. My sister's a lawyer. I trust her, and she's going to give me good advice. And she's an expert in law, but I don't have access to her all of the time, and I can't bombard her with questions all of the time. And with these LLMs, they are basically like having experts that are available all the time at the push of a button in your podcast or on your Phone and experts that are basically domain a specific, right? They're pretty good across a lot of domains. Of course they still make errors and hallucinate and all that, right? But that sort of distinction of being very, very low friction, easy access and being authoritative in the responses and domain a specific sort of makes them, I would say, orders of magnitude more integrated and capable of influencing and restructuring our thought patterns.
- [36:52]
David McRaney
And the fact that it uses language and that we're reading it in sentence form, to me feels like a big deal. Wikipedia is never also at the same time going like, hey, thanks for going to this article. It's a really good idea for you to be here. I bet you're working on something really important. And I'm fascinated by this, how, how I can get tricked by this and how it seems as if there's someone from on the other side who's like, yeah, turn the knobs of trickery, like, let it. Part of its function is to delude you and to keep using it more and more in this way seems weird.
- [37:24]
Dr. Stephen D. Shaw
But you can use a system prompt to reduce that dramatically if you want to, right? I mean, like, my chatgpt doesn't do that. I just tell you, like I have, you can develop a system prompt or find some online and you can really reduce that if you don't like that behavior. Like, I personally, I don't like and I don't find it useful for my work, right? And I need high fidelity critical feedback for my papers and my writing and, you know, my code. And so I use a system prompt to really reduce that. But with that said, I've got it so turned off that, you know, if I'm getting near ready to submit a paper or ready to, you know, send it somewhere, it will never tell me that it's good. And if I kept, I would just keep going, right? And I even, even today, Giddy and I were submitting a paper and I sent him screenshots of my ChatGPT's grade. You know, I said grade. This submission package for his payment gave me a B plus, you know, and then I, if I want to feel better about myself and I think the paper is good and it's ready to go, I'll just go over to my Gemini, which I don't have a system prompt on. That's like my sort of secondary LLM right now. And you know, Gemini, I sent Giddy the screenshot of this one. It said, this is an outstanding contribution to humanity. You know what I mean? The best thing ever. And so I actually use that to my advantage, because I want to be able to submit this thing at some point. And so I need a little bit of that positive feedback to say, okay, this is good, and I reaffirm that, and then I move on. But it's a very, very real thing, and a lot of people are not aware of that. And if you use like the free version of these LLMs, then you get a lot of that. And a lot of people, you know, maybe don't get a lot of positive feedback even in their lives in general. And so that can be very, very enticing. And I know Giddy's very interested in this idea of validation machines, which I think we might do some research on. You know, moving forward, I get the
- [39:17]
David McRaney
sense that, oh, well, we'll develop a literacy for this over time. Like, this is just the early days, and some of us are, are better at it than others. And some of us came, walked in a little more hesitant than other people. And we're using it for different reasons. Like, people who do a lot of creative work don't want to outsource the creative part of the creative work. And so they're going to be some of the first people to raise flags. And then it'll all just cascade and we'll learn together culturally, and we will develop a literacy for this thing, both for consumption of it. Like, we'll be like, that looks. That sounds like AI. That looks like AI. And then for the using of it. What are your thoughts on this? Is it. Are we going to have. Is this going to lead to a literacy and we're just sort of in the, in the middle phase of getting used to it, or is this something that we'll have to actually ask for? Like, we will have to adjust the systems to make them less potentially harmful in this way. Just any thoughts you have in that regard?
- [40:11]
Dr. Gideon Nave
Well, you know, I think for start, we know that the industry always goes faster than anything else than academia than policy. So I'm personally not that optimistic here. And some of the things, once we realize them, it's already too late. And in this case, there is also technology that is moving quicker. So it's not like the case of cigarettes that, okay, we ban cigarettes or we make cigarettes not as easily usable. And even in this case, you start to see all these vapes and like, things that go under the radar. There is constant kind of an arms race going on. And in this case, the arms race is between maybe us as people and the strongest companies in the world. So I'm personally not so Optimistic about developing illiteracy unless there is some alignment of incentives, I guess. But is there? I'm not sure there is. Like, the element of incentives maybe depends on the goodwill of certain people.
- [41:31]
David McRaney
I'm worried, especially after reading your work, that I am going to lie to myself. And there's going to be something that I think is an insight. There's going to be some, like, synthesis of concepts that I think is unique to my way of thinking about the world that bubbled up out of my work. That is really the result of my interaction with this machine that wouldn't have happened if I had not used the machine. This is leading to this weird question, this thing that I kept seeing come up in your paper. Not just that, the sort of passively receiving it and saying, I'm not going to question it and apply critical thinking to it. It's believing that it was your idea all along. That's the part that freaks me out the most, is thinking that it was your idea like it came out of your head. I did the prompt, it went to the thing, and then I read the thing, it came back, and I own this idea. I would love to hear your thoughts on this, Stephen. This not just really wrinkles in cognitive control or whatever, but just adopting its judgment as if it is yours, adopting its insight as if it was your insight. What do you have to say about this? What did you learn from all this and what are your thoughts on it in general?
- [42:46]
Dr. Stephen D. Shaw
Yeah, and that's, I would say, another core distinction between cognitive offloading and cognitive surrender. In the paper, we delineate those. Right. And with cognitive offloading, we say it's strategic delegation, and you are still maintaining that agency. You're choosing and being deliberate about which tasks are. And components are being offloaded. With cognitive surrender, we say that the agency is being Transferred to System 3, transferred to AI.
- [43:14]
Dr. Gideon Nave
Right?
- [43:14]
Dr. Stephen D. Shaw
And so the locus of control of decision making is in the artificial cognition rather than in your own mind. And we say that people adopt it as their own, and we look at some metacognition. So we ask about confidence, for example, how confident are you in your responses to these logic and reasoning problems? And we see that when people are engaging in cognitive surrender or when they have access to AI, their confidence goes up by about 10% is what we saw in the first study of the paper. And so that 10% increase in confidence. And we know because of the way we ran the experiment, that it was this artificial cognition that came up with the answer and the thinking and the reasoning involved. We know that that was 50% of the time was giving people incorrect information. And despite the fact that 50% of the time was giving them incorrect, correct information, they were more confident in their, in their answers. And so that's kind of an interesting and potentially risky or dangerous environment where you have a boost in confidence and a transfer of agency at the same time. It doesn't matter when you're answering logic and reasoning problems in an experimental study, but when you now move into high stakes contexts like legal context, medical context, education, then who is responsible when biases and errors from AI are moved into that work? Who's responsible when there are errors? When you're adopting something from artificial cognition, we have to use AI literacy or prompt engineering or whatever it is to try, or UX design to try to ensure that people in these high tech, high stakes situations are still taking responsibility for the outputs, taking responsibility for the decisions when we want them to. And another point that you were getting at there with education, when you start cognitive offloading, you start experimenting with AI and you say, oh, I could get it to do something for me. And then it ends up being quite good at that task. And then you can slowly slip into cognitive surrender, right? And then now you've got yourself in a situation where actually all of the writing is being done by AI and that like leads to, and we're seeing a lot of papers, leads to de skilling. And so we're actually right now like watching people. Cognitive surrender is the psychological mechanism. We're watching people de skill themselves. And you know, in a workplace environment, if we're de skilling ourselves, we no longer have those skills. And then what value are you to the company if you're just a prompt engineer taking outputs and giving them to the company? So that's on one side right now if you have a skill and you're de skilling yourself. But the bigger question is in education and learning, if people are, or our youth are engaging in cognitive surrender during the learning process, they may never acquire those cognitive capabilities or skills in the first place at all.
- [46:37]
David McRaney
I come from old school literary journalism. Like I went to journalism, I went to school for journalism. And you would read, you didn't just read the works of people, you read about their creative process a whole lot of the time. In all that world of like lauding and creativity and the writing process, whenever you read anyone talking about it, any famous author of any sort, from Baldwin to Hemingway to whatever, most of what they write about is the anguish sitting down and trying to extract something out of, out of the process and then they laud the process is where it comes from. And I know from my own personal experience, anything I've ever written that's ever been worth anything, I had no idea that was going to happen that day. It happened in the writing and I didn't even know I had those thoughts. I never attempted to articulate any of it. And it's surprising to the point that you get used to. If I sit here long enough and I work on this long enough, something will happen and there will be an insight and I'll chase it down. I know for me, I have noticed as pulling LLMs into the thing that feels threatening to that aspect of it. And then your paper comes along and I'm like, oh boy, it really is. So Mike, to turn this into a question of some sort, Gideon, what do we do about this based off just this early research? If you're going to give advice to people who are going to listen to this and be newly freaked out, like, what are our. What should we be doing as just users of this? And this seems scary in some way and it feels like I do have some control over all this, there's one option which is to never use these things ever again. And assuming that that's not going to be what people do, what should we be doing as users of these products?
- [48:21]
Dr. Gideon Nave
I wish I had an easy answer. I think that the first thing we can do is just be aware of it. Know that we have a tendency to do that and also be aware that we are working here. If we are, if we want to resist this tendency, we are working here against very strong forces like the ease of things, the fluency of things is something that we are designed to like and to follow. And we have created now a world where it is possible to in many ways stop thinking, we are in the risk of stopping to think. I always talk about how singularity is going to come not from the computers outsmarting us, but from us basically stopping to think, stopping to engage. I like the debate on education, but I have to say, you know, education is not just schools and not just childcare places. It's also parenting. It's also what the kids do in their free time. It's about creating habits for life. And we have created an environment now where the short term incentive is going to be to not think. Potentially. There used to be a strong incentive to think, but now, you know, I need, I'm going to pay with effort here. So just like we created an environment where it's possible to move from point a To B without walking, we kind of disincentivized walking. And then slowly, slowly, when this has become more and more possible, we start to see that actually there is also a downside that you don't see immediately. And once this is going to happen, it might be too late for some people, it might be too late for us as, as species. I don't know. I don't have unfortunately an easy one for you. One thing I can tell you is about creativity specifically. So I'm. Some of my research is also doing is in my teaching is also about creativity and AI. You know, creativity is a bit of a special case because creativity is to some degree defined by being novel and being unique and being special. Special. Now I'm going to be the last person that will tell you that AI cannot be creative and generate creative outputs. There are amazing cases like I don't know, even in the case of the game Go, that AI beat a human in it and, and did it with a wonderful new type of playing the game, changing a game that has been played for 3,000 years and, and led humans to think more creatively in this game. But if we have a tool that is available to all of us, by definition the use of this tool is making us less creative. We are becoming more generic. And in this sense, thinking from your own mind and basing it on your own experiences and on your own perception and on whatever data set that your brain is using that the LLM does not have access to, this is going to make you more creative. So I'm less worried about creativity. Creativity is kind of an ever moving goalpost where you shot and you made a goal. If you shoot to the same place again, you're not gonna score another goal. It's constantly moving. So I'm not that worried about creativity. But you know, critical thinking may be a muscle that we will stop using because in this case the incentives are not aligned maybe with making this extra effort because the AI is right a lot of times and when it's wrong, we're not even getting the feedback and we're not aware that it's wrong. So we can solve it with user interface and with maybe changing all sorts of cues, maybe having some things that will force people to take responsibility. I know that in law it is the case you're going to supply, submit a low case with citations to present presidents that do not really exist. You're gonna be punished for this pretty badly. So you know, that's one way of doing it. But in our day to day, and there's like just kind of whatever thought that comes through your head, why not just use the AI? It will be difficult. And I think we are, we're climbing a mountain here against tens of thousands, if not more years of the evolution of our brain.
- [52:52]
Dr. Stephen D. Shaw
You know the second half of the title of the paper is the rise of cognitive surrender. Right? And so we're seeing this happening experimentally right now. And I think we can see it in ourselves and in society.
- [53:08]
Dr. Gideon Nave
Right.
- [53:08]
Dr. Stephen D. Shaw
But this is only the beginning. That's I think the societal implication here. Right now we are constrained in the way that we interact with AI and interact with System 3. We use our phones and we type or we use our voices or computers. Phones will advance and we'll move on to the next medium and computers will advance, but the interactions between human and AI and the friction will decrease even further. And so if we're seeing cognitive of surrender now where we have these restrictions between physical boundaries, you can only imagine that that access and that lower friction will increase opportunities for cognitive surrender. I often think about the next thing is probably going to be some sort of AI glasses where you can have prompts on the inside of the glasses and you can also have prompt pre process stimuli coming in that you see that changes, you know, and gives everyone a unique view of the world. But maybe some people in those instances, right, Some products, right, they can do this syncofancy and they can feed you information on the inside and prompting. I think of instances where we might be in the street having a conversation with someone and you might not know whether they are actually thinking or even putting any effort into replying to you or they're just reading off of a prompt that has already been generated on the inside of their glasses and replying back to you. Or maybe people really want to tune out and when they go into a restaurant or they make decisions, those decisions are just dictated to them and surrendered and they just engage and make those decisions and, and it's definitely a sci fi type future. But if you think about this friction argument and how the cohabitation and the reduction of human and AI physical distance, we have to imagine that cognitive surrender is going to be ever more present.
- [55:16]
Dr. Gideon Nave
Yeah, I want to follow up on this in one way. So one thing that this study does is it takes a problem that is very, very complex. Like the question you asked me. Now I really don't have a good answer to it, but we can try to answer it empirically. What we have is a test bed here of a situation that we created a paradigm when we know, people exhibit this surrender and now we can bring up all sorts of potential cures or treatments and test it and see who are the people who are most vulnerable to it and so on. Like for example, we see the people who trust AI more are more vulnerable to it. Not surprising. People who are more intelligent also tend to be in fluid intelligence, tend to be more resistant to surrender. There are all sorts of individual differences factor. There are also situations when we are more vulnerable. I think that we don't want to put a blanket against AI and just have people stop using AI. What we want to be is more precise and more targeted and smart in our UX design. And this task is really giving, is really giving us some kind of a test bed where it's exactly what lab is about. It's a place where we can try things and then maybe bring them to the real world and learn insight. Learn what is the theory here? Probably friction is a big one. Like you can think of AI as something that chews your foot of information before you swallow it. And the less it chews choose for you. Maybe the more you have to do the work yourself, you will be maybe more resistant to relying on it. There is a issue of speed and accuracy trade off. Like, you know, when you, when you really need to be quick, when there is a competition, when you have tons of other tasks to do, you'll probably be more prone to stop doing it. And it does seem like in the real world now, AI is actually producing a lot of tasks for people to check it. It's something that people call is call AI brain fry. That they're kind of like just instead of generating stuff on their own or thinking intuitively, they're just checking the AI all the time. Kind of like workers on assembly line that look at an assembly assembly line of untaught thoughts produced by AI and trying to find what they are. So in this kind of situation, maybe you're actually more prone to surrender because you're overwhelmed and you have lots of opportunity cost. By the way, unto thoughts is a phrase that Pope Leo.
- [57:57]
David McRaney
I typed it immediately. I was like an assembly line of unthought thoughts is Pope Leo.
- [58:02]
Dr. Stephen D. Shaw
He has an official Vatican message. You can see it on their way. And he's got actually a very interesting one on AI and, and the quote I think is he's voicing or raising alarm bells that if we keep using AI too much, we might become passive consumers of unthought thoughts. Foreign.
- [58:46]
David McRaney
That is it for this episode of the you are not so smart podcast for links to everything we talked about. Head to you are not so smart.com or check the Show Notes right there in your podcast player. My name is David McCraney. I have been your your host. You can find my book How Minds Change wherever they put books on shelves and ship them in trucks. Details or@davidmcraney.com and I'll have all of that in the Show Notes as well right there in your podcast player. On my homepage davidmcraney.com you can find a roundtable video with a group of persuasion experts featured in the book talking all about it. You can read a sample chapter, download a discussion guide, sign up for the newsletter, read reviews, all sorts of things. For all the past episodes of this podcast. Go to SoundCloud, Apple Podcasts, Amazon Music, Audible, Spotify, or you are not so smart dot com. You can follow me on Twitter and threads and Instagram and Blue sky and everything else that's like that at David McCanney at symbol David McCreeney. Follow the ShowtSmart blog. We're also on Facebook. You are not so smart. And if you'd like to support this one person operation, no editors, no staff, just me. Go to patreon.com you are not so smart pitching in it. Any amount gets you the show ad free. But the higher amounts that'll get you posters and T shirts and signed books and other stuff. The opening music that is Clash by Caravan Palace. And if you really, really really want to support this show, the best way to do that? Just tell people about it. Either rate and comment on it on all these platforms or just tell somebody directly. Hey, you should check out this show. Point them to an episode that really meant something that really connected with you and check back in in about two weeks weeks for a fresh new episode. You're never just one thing.
- [61:11]
Dr. Stephen D. Shaw
You're the boss.
- [61:12]
David McRaney
Hey Google, when's my next meeting?
- [61:14]
Dr. Stephen D. Shaw
The athlete that class wrecks me and their mom. Everyone in the all new Mazda CX5
- [61:19]
David McRaney
more to move every side of you. Learn more@mazdausa.com Google is a trademark of Google LLC. Sequences shortened and simulated.