← Practical AI in Healthcare
Loading summary
Transcript71 lines
- [00:02]
A
Welcome to Practical AI in Healthcare, the podcast that cuts through the noise to spotlight real world solutions delivering real world value. From patient care to clinical research, from life sciences to patient engagement, we focus on what truly matters in healthcare today. No hype, no theory, just practical insights where AI is making a true impact. Dr. Steven Lapkoff and Dr. Leanne Rosenblitt are your hosts as we explore what's real and moving the needle in this exciting new domain. Welcome aboard and let's get to it.
- [00:32]
B
All right, usually on Practical and Healthcare, we spend a lot of time on the practical challenges of implementing AI in clinical settings. But underneath those challenges are some fundamental facts about how humans think reason interact with technology. Today, I'm joined by one of the foremost cognitive scientists in healthcare to help us understand why those foundations matter. Dr. Vimla Patel, a foundational figure in a field, has spent four decades studying how physicians reason, how they interact with technology, and what happens when the system design ignores human cognition. We're going to talk about what cognitive science can teach us about getting AI right in healthcare. But before we get to the exciting science, Vimla, your path here is a story in and of itself. You grew up on the opposite side of the world, literally, and you probably have a longer journey to where you are today than most. So your. Your journey here has both a geographic component, a cultural component, and an intellectual component. Tell us how a girl who was born in Fiji wound up in the northeast of the United States.
- [01:41]
C
I always wonder myself. I was born in Fiji with Indian parents and my father, with his wisdom decided to send me overseas to study. And New Zealand was the closest available place to go. He didn't want me to go too far. At that time, women from Fiji really didn't go out to study anywhere. We could stay at home. So I was one of the two women who went to New Zealand to study. And I was in the south of Ireland in Otago University and then to Otago Medical School. I'm hoping to do medicine, but my focus was really more on science. I liked biochemistry, physiology, microbiology a lot more than actually the actual practice of it. It didn't seem like my kind of thing. So I did undergraduate degree in biochemistry and microbiology and then got a job in Australia in University of Melbourne is teaching biochemistry to medical students at the same time enroll in a master's program to complete my biochemistry training. I was so excited about teaching them, but they were not so excited about learning what I had to say. And I learned very quickly that they really did not. I really taught what I Wanted not what they needed to have. And I found scientific foundation was lacking for training. We didn't know why we were teaching them. We just taught them what we thought was right. So I decided that I really need to drop this area and think about how to improve medical education, basically. And that brought me to Montreal in Canada. I was admitted to McGill University in Montreal and to education in Cognitive psychology and subsequently in cognitive Science to do a master's and PhD program all over again. And the detail the idea was and the study, basically we're all in the domain of medicine. So I worked very closely with the Department of Medicine together with the cognitive and educational psychology. And after I got my degree in 1982, when I graduated, I was, I was appointed had joint professorial appointment in Department of Medicine as well as in education and Cognitive Science, which meant that I was really set to use my grounded in medicine, yet use the theories and methods from education and cognitive psychology. And that really started me off into setting new kinds of studies I wanted to do. So I brought me right across other part of the world, from a warm beach area to beach bum to a really cold Montreal area with lots of snow. But it was fascinating. I love my journey through the thing.
- [04:46]
B
It's such an amazing journey. I want to probe a little bit why cognitive science specifically, what questions couldn't you answer without engaging medicine from a cognitive science perspective?
- [05:00]
C
I was looking for something that provided scientific foundation for education about learning. And what did they really need? I wanted to know what do doctors need? And what I'm teaching is not what they need and they're not motivated. So first of all, we taught them far too much science. They didn't need that much. It's interfered with what they were doing. They were not motivated. At the same time, I wanted to understand what do we know what expert doctors do so that we can and also what medical students do so that we can reach to show how students should be trained to approximate that expert and not to just teach them expert rules because they can't follow expert rules. The idea was. And then when I went to Department of Medicine and asked the center for Medical Education, can I work with you? And they said, but you people work here really need training in education and psychology. I said, really? So I went to the education psychology department and subsequently realized that I needed more than psychology and education and some background in linguistics, some background in computer science, something in philosophy and neuroscience, something overall that kind of comprised cognitive science, but theories and methods from it. And that was an exploratory study. My investigation to see can I do it. And sure enough, that's where I found my home and using those. And my dean was very, very helpful to me. He only appointed me, told me that, you know, you can do what you want in this department and in the center for Research in Medical Education and subsequently cognitive science as long as you meet with me once a month what practical things I want to do.
- [07:06]
B
So I love that you always had a practical angle where you're trying to answer a question of how to educate doctors better. And the way I heard you describe your theoretical orientation because your career spans so many interesting questions is that the animating question was always what is the nature of expertise in medicine? And I completely agree that cognitive science and cognitive psychology is a deeper lever on that than just thinking about education. So you know, a little bit self serving on my part. Right. As a, as a cognitive scientist. But let me kind of ask you about one of the most important findings in your career is that expert diagnosticians reason in a particular direction and those who er, tend to reason in the opposite direction. Most of our audience has never heard this. Right. This is in fact that whole vocabulary, that metaphor of reasoning in a particular direction is going to sound a little bit odd to their ears. So what I'd love us to do just the next 10 minutes is walk us through it the way you tell it and use the endocarditis case because it makes it vivid, right. We can tie it to an actual specific case and explain what does it mean to reason forward versus backward.
- [08:20]
C
In fact, interesting endocardis case that I used was became quite famous because a lot of people use the same one to compare the study data, you know, to see that. So idea was I so I built my lab over the next few decades after graduation and the idea was to really to study the process that led to the outcome, which is the outcome of accuracy. What did doctors and physicians do in the clinical world? How do they think? How do they, how do they make decisions? And we found out just asking them what they do isn't the way to do it because they tell you what they think they do, but not what they do. So we wanted to capture the process and we know that these process of reasoning, thinking, decision making are not visible and brain are not visible. Therefore you have to use a probe to be able to get to it. Cognitive science methods provided that probe and that such a probe means you ask them to think aloud as they are working through their thing and you record them and also give them sentence at a time. To make sure that they're thinking as they're going to make a mistake, they backtrack and you can use that. So that's what we did. And we developed work very closely with clinicians in the hospital. And the endocrinology case of bacterial endocarditis we developed with the help of a physician in the hospital endocrinologist. And that was a real case that was modified for our purpose. And so this case involved a 22 year old male who arrived in the emergency room with punctual marks on the arm and he had chills and he showed signs of fever. And subsequent examination showed he had two by six early diastolic murmur and he had eye hemorrhage in the eye. And so experts were asked to walk through this case. So they walked through, they said very little, they went very little detail. So eventually they kind of just said well it's definitely infection. And two by six, that is dialogue. Mamma tells me that it's an aortic involvement, probably iotic insufficiency and hemorrhage tells me there is a embolus or emboli in the eyes. And subsequently. And I would, he said my final diagnosis would be. And also the splint seemed to be normal and the fever was of very short duration. So I would say probably it's acute bacterial endocarditis with auric insufficiencies. And that was the correct diagnosis. So every subject and physician who got the correct diagnosis used it in a forward direct way.
- [11:31]
B
Um, and, and what you mean by the forward direct is they looked at the symptoms, they saw some kind of a pattern and they inferred from the pattern what the underlying causal system was that was producing the pattern. Right.
- [11:46]
C
What they did was they only used patient information, you know, fever, they showed signs of fever, they had chills, there was a hemorrhage in the eye and there was signs that showed 2 by 600 diastole at myrmo. So use only patient given information and draw very quickly in a set of inferences towards diagnosis. Inference meaning people like novices or people who are outside the domain will just take the patient information exactly as it was generated. Inference. For example, a 22 year old male was recently lost his job. Inference would be a young unemployed male, you know something. So they use inferences. A directionality of inferences move towards the diagnosis very quickly and not long. They did not go backwards towards explaining anything or justifying anything. But the people who used backward reasoning, meaning they went back to explain they weren't Quite sure. And this is also very common with physicians who are uncertain. So you might say okay, I think it's got a bacterial endocarditis with aortic insufficiency. But I'm not quite sure. Let me go backtrack and check against something else. That would be backward reasoning. So there would be more in the causal direction. So for example, an early medical student said I think it's the, the patient has bacteremia with valve involvement. So I'll have to know how do I explain the patient data. So he would say something like given that he has infection, a fever is probably an infection infection, he probably has fever and fatigue. That will probably explain it. And because he has bacteremia now, valve involvement will also describe the bloodshot, the hemorrhage in the eye. So what else will explain. So we'll go through each of the system and try and justify it's completely in the backward direction. So to move from hypothesis that he already had or bacteremia with valve involvement and going backward to justified based on patient data, physicians will never do good physician or expert will never do that. We'll use the indicator patient symptoms and indicator. For example you're taught in the medical school as the medical students would do. Infection causes fever by a series of steps in between. When patient arrives in the hospital it doesn't show infection, it shows signs of fever. So physicians only use what information is indicating immediately to be able to drive forward towards a diagnosis. That's why they're so fast, they're really quick to do that. But now or the experts outside the domain will have problem because they, they don't do the same way. For example, a psychiatrist working in the same endocrinology case which diagnosis she made was oh, it's, it's really a septic shock probably due to and taken from probably a drug use and probably because he was unemployed, he took drugs and it was all about society not taking care of people like that. And went on to explain lots of other so and back and forth, back and forth through the thing. But the diagnosis is very context related. Wasn't in the context but because they are psychiatrists after all fully blown physicians. So they do have a knowledge structure which is reasonably better. Yeah, so they had mixture of both. So it finally they did get the diagnosis but took a long time.
- [15:54]
B
It's a super interesting pattern and I know we're trying to summarize like decades of confirmatory research that really fan this out but if I were to summarize it in simple language and tell me which parts I'm getting wrong. Is that forward reason? I would say that forward reasoning rests on a structured existing knowledge base and extensive pattern recognition based on some sort of schemata. Of course, as a cost, it's error prone, right? So you see, psychiatrists are not going to get it right. They're going to make mistakes and have to backtrack and intermediate. And so as a result, because there's no fidelity to check, intermediate users have to use a mixture so they can't actually filter the signal from noise. But the implication is that backwards reasoning is what you need to use when you're learning, when you're not an expert in domain. And forward reasoning is something you can only use when you're sufficiently expertise so that the pattern matching is going to be right most of the time. Is that kind of a reasonable summary of.
- [16:55]
C
In fact, it was so good you should have done the podcast.
- [16:59]
B
Well, that's all I could manage. It would be a very short podcast.
- [17:04]
C
Is that this is a hallmark of expertise actually. And that is very true from other domains as well. Physics, chess, they have series of patterns and those patterns are very, very important. So these patterns, the kinds of schemata they develop, as you said, and that helps them screen critical information from not so critical information. This is what they develop with expertise. And these kinds of reasoning can never be taught. They have to be ingrained in the environment where they learn these patterns. But you can help them by providing series of patterns, you know, kind of whole pattern. So this is definitely recognizing patterns that would be very related to kinds of work they. So if they have a 10 or 12 kinds of patterns, they can quickly master to see which one it matches best and what doesn't match. Now this is very similar to what Daniel Kahneman found System 1 and System 2. System 1 is related to very quick, efficient recognition of something, whatever. System two is very slow and analytical and detail goes into causal area where system one does not. It's a heuristic nature. This is similar to forward and backward reasoning. The concept of Daniel Kahneman of System 1 was related Psychological concept related to speed, how quickly you can do things slowly. Whereas the forward backward reasoning is related to a process of the inference flow in reasoning. But basically it's not unrelated in some way.
- [18:54]
B
Basically it feels very close. Your claim is much more specific to the acquisition of expert knowledge where I think the Kahneman model tends to be much more general. But certainly the features of rapid pattern recognition that occurs without noticeable processing delays versus effortful processing, that sort of Applies rules and is hypothetical deductive in a sense. Right. You know, you. It feels like it's very matched and I, I'm not expert in either one, but it's sort of conveying that same sense to me. But it's a fascinating and provocative claim.
- [19:30]
C
Right.
- [19:31]
B
That in a way what you're saying is that forward reasoning cannot be taught and backwards reasoning can. So we have to start with backwards reasoning and build experts who can actually apply forward reasoning. I just find it fascinating.
- [19:43]
C
You have to. And see, we never realized the implications of such work at that time. First paper was published in culinary science in 1986, and we were more interested in identifying experts. And my dean wanted to know, how should we change our training program for medical students? Practicality, but more on the education side and training. And we did change a lot of medical education at McGill and other places based on that. So it did impact. But there was no technology at that time.
- [20:18]
B
Yeah, but wait, so now let's build the connection to not just practical education, but practical AI implementation. So if that's how expert clinicians reason. Right. If it's, if it's, you know, that's a huge differentiator between forwards and backwards reasoning, what are we getting wrong when we're building AI tools for them?
- [20:40]
C
The AI, the AI systems assume clinicians reason in system two by backward reasoning, therefore set up that way.
- [20:55]
B
So we assume when we build our tools that people are going to derive and do sort of systematic, effortful inference from some sort of principles. And all we have to do is give them lots and lots of information to make those inferences from. I think that's true. I mean, I think by and large that is the shape of the tools that people are delivering.
- [21:15]
C
I assume that before they're experts, they'll sort it out. You know, thing is that that's not how people work in the clinical world. They actually use for directed reasoning at most times. And second, if you give them too much information like I just described, it interferes with the utility of actual performance. So the first just, it's really broad reasoning is related to performance. And if you want them to perform quickly, efficiently, like system one does, you really need to go provide them with lots of patterns of, you know, data for it. Not, not kind of lots of details about causal reasoning, but it should be allowed. You should be able to activate underlying knowledge when they need it. Because if they can't understand saying, why does the patient have, you know, something else that I don't understand? Because this patient really shouldn't have that Given the diagnosis, you need to be able to trigger a backward reasoning. So they should be able to do that. So this is where the system does not do that so well, such a good job of it, you know.
- [22:25]
B
So let's give builders one concrete design rule that if Patel was in charge of building systems that you would say you have to keep this in mind.
- [22:37]
C
Yeah. So I would basically say that people often use both sides for listening to science. So the thing is that I would say highlight more patterns for a start. Don't give anything else. Embed the AI system into EHRs and then making sure that the physicians it's disembedded where physicians normally look and search not some create an additional rules and additional things. The other thing we found that when physicians change context the forward reasoning falls apart. Don't minimize the cognitive switching from one task to another. And this is very important. Don't force them to switch. So reduce that kind of thing and don't give them strings and strings of differential diagnosis. Minimize it as much as possible. If you have to give anything, give some summary of some context related information. Certainly do not give long list of clinicians. So that way you diagnose it such that the expert expertise will be preserved expert performance and the same time there will be less clinician burnout.
- [24:01]
B
So I love that. And it's really a very strong design implication that you cannot build a one size fits all interfaces. An interface that will support a novice in a domain well, will be absolutely destructive to an expert. It will make them a novice in using that interface and they will fall back into novice performance. Which is a fascinating counterintuitive thing. Right. I mean I would say designers have to understand it and all of us who are out there building actual AI systems really need to keep keep that in mind. I mean, does that sound like a right principle to you?
- [24:37]
C
Absolutely, absolutely. I think the designers have to, but they think they usually most often work on the thing that we have to build systems based on what physicians do. But what physicians do is not what you see with your eye but what you have process. You can't just do quantitative studies and say well this is what they do. You know, problem based learning. Neuros, when they said, oh, they don't use any medical science knowledge, they only use clinical indicators. So we don't need to teach them anything that is wrong. Because when you look at the probe further you find they do use only when they need it.
- [25:21]
B
Yeah. So you've made, you know, a really good case that what builders need to do is Augment expert pattern recognition. And that rather than trying to think for the clinicians, they need to collaborate with academia and with practicing clinicians. I just love that insight. But I want to take us to in a way like the opposite case, right? So you investigated one real case where very sick patient got a massive potassium overdose. And the striking finding, you know, from your team was that the clinicians did everything right. Tell us that story.
- [25:59]
C
It's a very interesting story, in fact that that study actually highlights that a real problem with our decision support systems that they provide and probably a clear example. So the one of the hospital adverse event committee identified a problem and I happened to be on that committee that a patient was 85 year old patient arrived in ICU with sort of. I think that was the arrived in ICU with the admitted with ICU with the septic shock and respiratory failure. And he was. This patient was given over a total of three days, was given 300 and excuse me, 316 milli equivalents of potassium chloride. And that if anybody knows anything about potassium chloride, it's a very high dose. And so this after. So it went on for three days. And not only that, it was the errors were supposedly made by series of healthcare providers. So I was given the task of finding out what exactly went wrong and how can we train our doctors and residents to a better job. Assumption was all the mistakes were done by the residents, nothing to do with the system. So with the help of my at that time, my doctoral student Jan Horski, who led the study, we analyzed the log file. So we knew in every detail in the log when the patient was admitted and what happened from the time first resident came along and how much dose is the proteosin chloride given? What was the dose and what time it was given. Was the medication active or was it discontinued or inactive so we could follow that trace. So we found that resident A when he arrived, he found the patient in the lab data showed the patient was hypokalemic, okay. So really needed to give potassium. So gives the orders a injection of 40 milli equivalents of potassium chloride. So that's fine. And that was a correct decision, dude. So he clicked the system on to say order 40 milli equivalent system didn't react. So discontinued. That as a right decision again and clicked again and then it didn't react. So it discontinued, which was the right decision. And then he ordered a injection of 80 milli equivalents of potassium chloride. And then, pardon me, 60 milliliters equivalents of potassium chloride. So that Was done. So when subsequently we found that what had happened is the first order in the EHR showed that all data was still there. They had not been updated because all the system dynamically updates. Current data was not there. So the first day, first when he clicked the first time, it was discontinued, as he said. Second one was not discontinued. And that compounded with the additional injection together, he was already receiving far too much. And later on that afternoon there was an order because he still showed he had hypokalemia. So the lab data was very far behind. It didn't record the latest again one of the biggest problems. So ordered another 8 milli equivalents in a 1 liter of fluid, 1 liter of saline, the glucose saline one that was. It was so 80 ML equivalents of 1 liter. But what had happened is the system did not stop at one liter. The system had a default of seven days, which.
- [30:25]
B
Oh man.
- [30:27]
C
That continued along until three clinical rounds, until the next person came along. So you can imagine how much it was, how much of the thing they were given, how much potassium chloride. Okay, so physician made the right decision at all times. The system did not support the decision they had. And so the next day, when the second resident came on board again, the data showed hypokalemia. He couldn't understand why was it that it was still hypokalemic. So ordered an additional injection with the saline flow of 60 millie equivalents, which was the bolus to be given. It wasn't easy to calculate the bolus against the fluid, which was by the volume on the saline. So that wasn't quite easy visible. So you had to calculate all that and then audit it. And then finally, third day they realized what had happened. So based on all that, eventually the end it was 316 plus milliequivalents of potassium chloride. It was administered over the period of third three consecutive days by three or more residents. And finally found out and patient was treated, however, that our task was to alert the system builders that they had to change the system. And second, introduce a training program.
- [32:04]
B
So then what would be the design change that would have caught it? Right. Let's think about. And what ways would you recommend? What would you recommend should have changed in the interface and the rest of the system that this would not occur.
- [32:17]
A
Right.
- [32:17]
B
I mean, I'm not a clinician, but it sounds like an absurdly high dose that's like one or two orders of magnitude higher than what the patient should have received. You know, obviously a series of very rational decisions and reasonable decisions by the clinicians that interacted with the system to produce a really bad result. What's the interface change or sorry, not just the interface, I. Cause part of it is a timing issue. But what's the design change?
- [32:44]
C
First of all, I don't think anybody ever understood that you have to study the interaction of the doctor with the system, actually do what they require and how the system responds. So it's really evaluation of the human computer interaction here and saying what system do they really do? What do they press, what kind of response do they get? And that was very clear. It was just quite clear that the way they do things, way they calculate, way they think was not aligned the human cognition was not aligned with the way they think and work and the way the system was built anything. So systems should have many things. For example, they should have a checkpoint, for example, it should have a stop by 1 liter, not by going completely default flow of 7 days. It should never happen. There should always be an alternative check for that never happen. And the system overall data flow that if they had continuously had all data, somebody should have checked or somebody should have a checkpoint or multiple people working saying, it doesn't sound right. Yeah, some information was given. It doesn't sound right. Which means a system should alert. I mean working constantly. So the system should alert. Watch out for this thing. This is not right. Something's wrong or alert. Even a certain potassium level it reaches, for example, it's a dangerous thing that should definitely be there and that was not there. And that's absolutely critical to have so more of these things. And the important thing is that if they don't fix all of those things and we identified a lot of these errors, they don't fix that thing. Then no matter how smart solutions are, they will make some really dangerous mistakes.
- [34:45]
B
Yeah. I think that you're raising a point that's absolutely critical to hear for all the designers, that point of that of mutual literacy. Right. That you need to train a clinicians to understand the system and we need to train the designers to understand how clinical cognition and workflows actually work. And the safety needs both halves. And you know, I'm going to quote back to you some something you said to me earlier in a previous conversation is that when technology ignores how clinicians think and work, it doesn't just fail. It quietly sets the stage for smart people to make dangerous mistakes. I think that nails it. I'm like, yeah, that's the way we have to think about it. So I want to. This was so interesting.
- [35:31]
C
Right.
- [35:32]
B
And I kind of love the contrast between those cases. But let's end on what worries you, looking forward, specifically about AI, modern AI, at least the one we're seeing today, that's designed for consumer use and some for clinician use, speaks with a lot of confidence. It doesn't hesitate. What does that do to clinicians over time?
- [35:56]
C
Well, you're already familiar with it now, so excuse me. Decision support systems that we use, because they're a support, as they say, means physicians offload a lot more of that information onto the technology more and more so that they become the expert decision makers. So more and more towards that rather than as a support. And the problem is that when they start to do that, these machines inside of AI that are not always correct and second is that when they take this response, when they're uncertain about something because they are so eloquent in the way it's presented to you, they don't stop when there's uncertainty. They kind of still present it as if it's a reality. And as human beings, we don't do that. When you have certain things that doesn't jive with you, you stop and think about it. This does not. So that means you offload a lot of the information to AI and you do not think anymore. So all these judgmental skills that are necessary in today's society, definitely necessary, they will all be lost because if you don't use it, you lose it. You know, it's absolutely important these kinds of skills. I mean the labor force, you're going to need people when technology is not available to make a quick judgment call on many things in emergency situation. But this is not what happens. So I think this is going to be, is really quietly eroding the human capabilities, you know, and then it's a critical skill that is required for independent judgment. It's not there. So I would really worry about that. And this will really mean that AI interfaces with the built in features that focus clinicians to pause and verify. The AI have to build in system. Every time they're uncertain, there's got to be a pause and able to checkpoint to say do you really want to accept this advice?
- [38:12]
B
You know, so love the idea of built in friction points and you know, as, as ways to structure around human cognitive limitations and features help us understand what would a pause and verify interface actually look like.
- [38:29]
C
I would say whenever there is an uncertainty in something that it should not just give a eloquent response, but the system should build in and say here is an uncertain. We want to check that point.
- [38:47]
B
Yeah. So you want to replicate some of the uncertainty and certainty gauging signals that humans send to each other.
- [38:55]
A
Right.
- [38:55]
B
Like when I'm not sure about, you know, something, I'll say, I'll pause, I'll hesitate, I'll send you strong signals or at least if I'm, you know, epistemically humble. I know people who don't do that. We, we all do. But you know, we in, in constructive interactions, we shouldn't. So it's the idea would be to have an interface that somehow that replicates that in our AI cognitive scaffolding as well.
- [39:20]
C
Sure. That it's possible to do, but how it does it, I have no idea. But to be able to do it, because you've got to stop, you can't offload. Over time, you'll be offloading more and more information completely to the system because it's easy to do and it's quick and it seems reasonable and it seems to most often give you correct answer, but not always. And we're talking about human lives here. So you really should think about practical point of view is what we are using. Because our studies spanned many years. We were able to do the studies when no technology is available to what is available currently. So able to compare over the time. I think this thinking skills and judgmental skills, higher order judgment skill, which we don't teach even in medical school. This is going to be a problem unless you really force people to do it. But even through technology built in.
- [40:21]
B
So given what you are seeing now and from the perspective of that depth of understanding that you built up, how would you train residents differently today in an AI saturated clinic?
- [40:35]
C
First of all, they would have to begin using AI early. This is they don't stand alone thing. You know, it's part of the we work with other humans in a kind of distributed way. You know, we work with other humans and we work with machines. Now we talk and we have inclinate intonations in the way we talk, the way we say something. Our language tells us how I feel about you or what it is. When you have this and you have a system, it doesn't. So a system needs to be much more flexible in this way way that you can work. But they have to be taught very early that the system will not give you everything you need. You need to have a human skill. And therefore judgment skill and higher order reasoning skills have to be tested without the use of technology someplace or other. Because if you don't do that means residents should be trained to be able to say if Something happened. And if you potential situation power goes down, you don't have anything. You still have to make decisions. You have to be trained in it and also to train what system means, what system does to you. In other words, if they, if you press, click a button and press active order and if it doesn't respond, don't assume that it didn't go through. You know.
- [42:01]
B
Yeah, I'm going to quote back to you something you said to me because I think I just thought the phrasing was really evocative that we can no longer view AI as a passive tool. They influence our thinking as much as we shape their behavior. And I think that understanding now, at least today only developed with expertise, much like being a radiologist or being a good surgeon. And there's just this vast difference that I see in the IT space between people who've become, who've immersed themselves and have become relatively proficient users. I don't think any of us are experts, is moving too fast to become experts. The people who are relatively proficient and those who like, just haven't gotten there yet or haven't started. And I keep hearing these comments like, oh, it doesn't do that. Right. And you know, and you know, as a, as a relatively proficient user, you want to say, dude, you just don't know how to use the power tool. Like, it's like somebody gave me, you know, a lathe and I'm not a machinist. Machinist, right. And said here. And I try to use it and I'm like, it just hurts my hand and it breaks the piece of wood that I stick in there. I go, this thing doesn't, doesn't do anything. It's like you don't know how to use it. You got to practice it a little bit, maybe learn, spend some time. And I think in one, there's a sense in which the modern the chat interface misled a lot of us into thinking, oh, this is a chat bot, right. We just talk to it. And it's a much more powerful tool. It's a language layer on top of some other important technologies that allows you to create really powerful automation stacks. But developing intuitions about what works, what's easy, what's hard, really takes like hands on grit. So I, and I really think that the cognitive science perspective is a very, very powerful way of addressing it. And I wonder if you could tie it together to your perspective on sort of the kahneman Think system 1, system 2, thinking what is it that we have to understand and know in order to design around these really powerful new automation stacks.
- [44:05]
C
Automation stuff is not exactly my thing.
- [44:08]
B
But think about the humans around it. Right. What I'm asking you is like what does the human have to understand right. In order to work in this environment?
- [44:17]
C
Well first of all they are there to support you for a stop.
- [44:22]
B
Right.
- [44:23]
C
You can offload the things that you. It's easy to do but you don't want to do because it interferes with your real task you want to do. So you can offload those systems to them but don't let it completely take over your the way you think and where you do things completely. But things that matter your priority. But offer load information so that you don't have coverage overload. And because you can do that multiple things and the system should not at the same time should not mimic expert behavior and become an expert but be more Augment. Augment.
- [45:05]
B
Yeah. Right.
- [45:07]
C
Like the augmenting so that they. It's. It's. It's really the augmentation that's critical on this part here. And then so the design of that system that I would like to use for example I want that is trained by the human centered AI centered what my tasks are. It's a generic what I do and how much can I let a lot of things I do it my cell phone now that I really don't remember any of the phone numbers. I don't remember anything in the emergency. If I ever needed it, I wouldn't have it because. So in the healthcare environment you want to make sure the skills that you need some of the skills to bear that you need if something fails in a surgical environment, for example robotic surgeries use of meds is very common now they come in but fails. You have to step in. You have that skill. So augmentation is the way to go. Not decision maker completely in that sense. And human centered AI that means it has to be done correctly. This is where the collaboration between clinicians and real collaboration they become co authors on your work. Not just people collaboration between that industry designers together and work on that system. I know it's labor intensive to do some of this thing. Industry is not always willing to because it is labor intensive. That means time consuming. But we have to do the right thing. You know, design things that are that. That work. And you'll be able to and understand that how you represent the world around you. How do you represent clinic patient problem will subsequently lead to how you make decisions about a patient. Yeah.
- [47:02]
B
So I love the. The human centered design focus and I've had the Privilege of working with some human centered design folks through the DCI network. You know, one of my favorite things you said that connected things up for me sort of to back to system one, system two, light thinking is that one thing we need to understand when we are in order to build effective systems, in order to do good human centered design is when to think fast, when to think slow, and when to think for yourself. And I can't wrap it up any better than that. So I just wanted, you know, just wanted to land that one, attribute it to you, because I'm stealing it, but I'm borrowing from now on.
- [47:41]
C
Exactly. Back of my mind is, you know, think fast and when slow and when you think, you know, independent of that, to yourself.
- [47:49]
B
Yeah.
- [47:50]
C
You know, that really ties into Kahneman and Tversky, Kahneman's work. You know, and the other thing is that earlier work, you know, you tried to put both forward and backward chaining in the same rule, but didn't work because it caused perturbations in the system. So now they've so much better system. You should be able to do something so that you can quickly stop and go analytic, go backwards, go system two when you want to, and quickly. As soon as you have built a certain schema, you should be able to move forward into system one and move forward with doing it. So it should be rapid succession of those things, you know, so that you are not always just doing fast, but you also have time to go slow, to go back to it and keep the flow of the inferences towards the reasoning direction you want.
- [48:42]
B
You know, it's great advice. Let me. Let's see if we can land on it, even on a really practical angle, because I think that was brilliant but conceptual. Say you're in an elevator with a health system CEO or an AI health, health AI entrepreneur, and you get 30 seconds to tell them about what they should know about cognitive science as it applies to AI and healthcare. What's the elevator pitch?
- [49:12]
C
That's a good point. What's the elevator. I never thought about an elevator pitch,
- [49:15]
B
but, oh, you don't go to a lot of entrepreneurial meetings. I'm pulling you out of your natural environment and say, like, you could do it.
- [49:25]
C
I would just basically say, oh, you CEO of. Which company are you CEO of? Oh, what system do you build? Oh, and I would love to come and look at it one day and see what you're doing. I'd be really excited and try and get to see what they're exactly doing and slowly introduce the idea of saying look, you know, this could really benefit from this. Studies could really benefit. We found this. You're thinking somebody may be interested, maybe you can collaborate. My pitch would be to get inside to see what they're exactly doing. Of course it's not always transparent to you, but at least get them to understand. Because if we just say, you know, oh, I'm a cognitive scientist, they will not be interested. Get myself interested in their work and what they are doing will be my priority. And then some students over to see what they're doing. Add some bright and brilliant students and graduate students who I can send over and look in to see what they're doing and see how we can recommend things they can do better.
- [50:30]
B
By the way, I just wanted just to indicate the footprint you've led industry. I keep writing to your students everywhere, like of the people you've talked about. Jan Horsky, I serve with him on an advisory board for SUNY Downstate. Dave Kaufman, he and I did some work together years ago and he's, you know, they're both fabulous. So you're leaving a wonderful legacy of really wonderful people who are advancing this. I'm much more bold. So let me just say what I, you know, I, as an entrepreneur and a, you know, a former cognitive science academic, it's been too long. But what I would probably say is you can't build what you're building without understanding how humans think about it.
- [51:09]
C
Absolutely no question.
- [51:12]
B
And we don't yet understand how humans think very well. So you're gonna have to bring somebody in to figure that out empirically.
- [51:20]
C
You really have to get them interested in showing interest in their work rather than trying to get them to interest in your work, because it's not going to work that way.
- [51:30]
B
Well, you're right in terms of long term interest. But in order to hook them, it's going to have to be a fear, uncertainty and doubt pitch. Right? Because you got 30 seconds. You don't have time to, you know, you, you literally just have, have to tell them honestly, which I think is true. You will fail on quality because you cannot build what you're building without understanding this piece. So forgive me for being bold. I mean, I'm, I'm not. You're. You, you've done your things your way and you're very successful. I'm, I'm just telling you how, you know, a guy who's used to pitching things would do it.
- [52:02]
C
I would love to do that, except I don't think many CEOs are going to accept that too.
- [52:05]
B
Well, probably, you know, you got to. It gets you a foot in the door and gets you to the next meeting. So, you know, so I thought this was just, you know, thank you. This was just a fabulous conversation, such a delight to have you. I mean, we talked about two cases that give us the same lesson but from opposite sides. You know, build with clinical cognition, support forward reasoning and you preserve expertise. Ignore it. The potassium chloride case and you and you set up smart people to fail miserably to the detriments of patients and to trust. Vimla. Before we wrap, tell us where our audience can find you and learn more about your work.
- [52:45]
C
I'm at Columbia University, so my Columbia email vimla Patelolumbia. Edu you'll find me. And all the studies that we have done over the years, they're all published because they're all available in literature and I have to just send you any one of those. In fact, over the years, people have rebuttal my work and they didn't like it. We rebuttled again and subsequently. But they're all in literature. So easily accessible and I'm easily accessible on the web. Just Google me and you'll find me somewhere.
- [53:20]
B
Well, wonderful. We were so fortunate to have you for this hour to have this delightful conversation, and I hope you know, we can bring you back some other time. And I just want to thank you again for joining us and thank our listeners for tuning in and we hope to see all of you next time on Practical AI in Healthcare.
- [53:46]
A
Thank you for joining us this week on Practical AI Healthcare. If you're ready to go beyond buzzwords and hype and explore how AI is truly transforming healthcare, stay tuned for more conversations that get us to what works. Until next time, stay practice.