← Practical AI in Healthcare
Practical AI in Healthcare
S1, E31 - Reflections 4: What Does the Infrastructure Actually Look Like?
00:51:35
Loading summary
Transcript83 lines
- [00:02]
A
Welcome to Practical AI in Healthcare, the podcast that cuts through the noise to spotlight real world solutions delivering real world value. From patient care to clinical research, from life sciences to patient engagement, we focus on what truly matters in healthcare today. No hype, no theory, just practical insights where AI is making a true impact. Dr. Steven Lapkoff and Dr. Leanne Rosenblitt are your hosts as we explore what's real and moving the needle in this exciting new domain. Welcome aboard, and let's get to it. Hello and welcome to this week's edition of Practical AI and Healthcare. I'm Dr. Stephen Labkoff.
- [00:39]
B
And I am Leon Rosenblit. Today it's just the two of us again, Steve, for our fourth Reflections episode.
- [00:46]
A
Right. And so for those of you who are new to these, every five or six episodes, Leon and I step back and we try to synthesize the themes that are emerging, the things that we're hearing that are coming out of these conversations. It's not really to summarize the conversations as much as it is to sort of see what picture forms when you put all of it side by side. And, you know, at this time, something really clicked, at least for me, when we sat down to do the prep. And I was going to go through. As I was going through episodes 26 to 30, you know, we went through the following, folks. We had Giovanni Donatelli from the language group on language Access. We had Charlie Harp from Clinical Architecture talking about data quality. We had Adam Bloom from Cancerbot talking about clinical trial matching. Sashi Shankar from Novelia who was talking about patient data. And Amy Price who's a professor up at Dartmouth, who was talking about participatory medicine. And these are like completely different domains on the surface. However, when we thought through it, I think we came up with the. And we agree that they all kind of point independently to the same place.
- [01:52]
B
Yeah. And the place they arrived at is that AI works. And that's no longer the interesting question. Right. Does AI work? The interesting question is everything around it. The data, the human roles, the patient relationships, the regulatory conditions. Every one of those guests had a working AI system, and every one of them spent most of their time and energy talking about the infrastructure that makes it useful.
- [02:17]
A
Yeah. Which was different than what we heard in earlier blocks. And, you know, in earlier discussions that we've had, we were talking about. We were arguing that if AI was actually work, what could it do?
- [02:28]
B
Right.
- [02:28]
A
You know, in block three, we talked about Josh Gileras, about revenue cycle management and the Proof case, and we talked about Alvin Liu's sustainability problem down at Johns Hopkins. But in this block, nobody was actually arguing about whether the AI was working. The AI was working. Now they're talking about the plumbing and talking about making sure that everything's working underneath to ensure that they're getting good output.
- [02:55]
B
I'm starting to suspect, Steve that's what industry maturation actually looks like. It's not Sam Altman running naked down the street yelling Eureka. With soap dripping off of his torso. And it's not a hockey stick curve. It's just a quiet shift in what the conversation is about. Shifting from, is this thing going to work or is it working? To what does the system need to look like for it to matter?
- [03:22]
A
So that's the frame for today. Let's start talking about the foundations and literally, Charlie Harp and data quality.
- [03:29]
B
So Charlie's been building health data infrastructure for decades. I mean, we've known him around the industry for a while. He founded Clinical Architecture, which makes tools for the healthcare terminology and data quality. And his core message was almost uncomfortably simple. If your data is bad, your AI will be bad, but faster. So he introduced, among other things, his picky framework. And I love the acronym. It's P I Q I. But it's be picky about your data and has four measurable dimensions of quality. And the number that struck with me, that stuck with me, is lab data quality across healthcare averaged about 70% against USCDI standards. That means 30% of the data feeding the AI systems is just wrong, missing, or inconsistent. And he has this great line that we stole, and I'm going to keep using it. And I had to draw a cartoon about it, like, we're building pipes, but we haven't decided if there are sewer pipes or water pipes yet. What I found striking is the timing argument. AI didn't create this problem. Decades of building focused data systems did. Data only improves when it's put to use and when people iterate on data quality. But the AI made it visible because the AI tries to use the data for something else besides billing, not just storage. And then regulation caught up. We had Section 1557 enforcement, which got stricter in 2025. And that made data quality a boardroom problem, not just a data scientist being annoying problem. So the business case for fixing data quality wasn't AI, it was compliance.
- [05:01]
A
Yeah, and I've been dealing with bad data most of my clinical career. EHR notes went from being so so to being full of copy and paste. Lab values are sometimes coded in, you know, not correctly allergy lists. Oh, my God. They can Be a mess. You can be allergic to something if somebody uttered it the wrong way, and then it gets listed as an allergy and it sticks with you for years and people tend not to update it. So an allergy list, or even a problem list for that matter, can dog you as a patient for, you know, a decade or more until somebody fixes it. And those are fundamental problems in the data. But AI doesn't really deal with it. It takes whatever's there as a fundamental. And what has to happen is that data has to be right. If the data is not right and you're feeding it bad allergy information or bad problem list, that's a significant challenge because the system doesn't know any better.
- [06:00]
B
I would refine it just a little bit, Steve. I mean, you and I have talked about data quality in our registry work a lot and I keep coming back to the concept that data quality is relative to the use case. Right. So I think folks in epidemiology understand this really well. Right. It's, you know, the data use drives, drives data value and it drives data quality metrics that you apply. So I think the difference you're seeing with AI, it's expanding the space of possible data quality uses beyond what we were doing before. That's my take on why we're seeing all of a sudden this is a visible problem.
- [06:39]
A
Well, it's a visible problem for sure. But it also gets to the point where is data quality also an explainability issue? If you can't trust the inputs, how do you trust the explanations of the outputs? That's I think, the real crux of the problem. If you're getting bad information in, in the model, what's that going to say about how the model is being used and what it's producing in terms of what, let's call it clinical decision support.
- [07:06]
B
Don't you. There's accuracy is one dimension of data quality for sure. And the question I would pose is, is it intrinsic dimension or is it an extrinsic dimension there? There are things about data quality that can be use case agnostic. Right. Like things like completeness, is it there? Right. You know, or provenance, does it do. Can we trace back where it came from? So would define. I would be careful about lumping trustworthiness and with data quality, I suspect they're a little bit separable. But I think the concept there that's related is provenance. Do we know where this came from? Can we trace down how this value came to be and the chain of custody that allowed you to Derive it. I think if you can't, then you're just making stuff up. And whether you're using fancy AI or old fashioned Excel spreadsheets, your results are going to be just as suspect.
- [08:06]
A
Well, the picky framework sounds like, as we listen to Charlie talk about it, sounds so logical. Sounds like it should have existed 20 years ago. I don't understand why every health system out there isn't using measurable data quality metrics. Is it just that nobody needed them before AI, or do you think because as you said, it's use case specific and now the use case seems to really demand a higher degree of fidelity. At least I think so. And listening to Charlie, I would think that's the, that's the use case and why that matters.
- [08:37]
B
Right. I kind of had the same reaction as you, which is like, duh, why, why aren't we, why don't we have this? Right? And I, I, I have a view on why we don't have it. And it's twofold. One is when you see in practice is the data quality improves when the data is used for a business purpose.
- [09:00]
A
Right.
- [09:01]
B
And you know, if you need to generate a report that gets presented to somebody, that's when the data starts improving. When you need to bill somebody and use the data, that's when the data starts improving. So trying to come up with data quality in the abstract is always a very difficult exercise. What I think Charlie's defining the reason his picky framework is going to get leverage and I'm strongly supportive of his work because I know you are. Absolutely. Is that the use that he's thinking about is data interchange. There's now a greater opportunity for data to be used for aggregation and exchange between institutions. And I think having a data quality metric and a set of data quality metrics that focus on that and say, yeah, this data is sort of usable, we know what it's usable is actually valuable. The other thing it required is as often as the case in it is development of underlying infrastructure. He's, he is assessing data quality relative to uscdi. Until you had uscdi, you actually couldn't say what the data quality was. Right. This is, you know, it's a chicken and egg problem.
- [10:07]
A
So yeah, I'm not sure. I think there were elements of it that could have been done.
- [10:11]
B
Yeah, but how would you get people to agree? Right?
- [10:14]
A
Yes, that's exactly right.
- [10:16]
B
It's a coordination problem. Right. You have to agree on what you, what the hell you're measuring to say, well, are you measuring at high quality? Well, UCI at least gives you a spreadsheet and says, you got to have these, these data elements and we can assess them and say, are they complete, are they consistent, do they fall within range, et cetera. So I. That's my guess about why what Charlie's doing would have been really premature if we try to do it globally, you know, at a broad scale a few years ago. But I think it's a. It's a very timely initiative and I'm really excited that it's moving forward. So I'd say Charlie Harp showed us the foundation your data is. If your data isn't right, AI amplifies the problem. Right? Just, you know, your AI amplifies the good and the bad properties of your data system. And here's the next question, though. Even with good data, does the AI model itself do the job? And Adam Blum, I think, had a surprising answer for us.
- [11:13]
A
Yeah, so Adam is the serial entrepreneur who has actually had several very lucrative exits. He's a really smart guy, really nice guy, and he actually turns out to be a cancer survivor. And he speaks about this very publicly. When he had to deal with the challenge of being diagnosed with follicular lymphoma, he built Cancerbot. Why did he build Cancerbot? Because he personally had a stake in the game in trying to find clinical trials that would be good matches for him. And he got really frustrated that he was having a really difficult time doing that. So he dug in. He applied his own information with a large language model to look at the eligibility criteria, and he was getting results that were like 60% of accuracy. And having worked in the clinical trial space and at big pharma companies, I can tell you this has been a really core problem with getting patients into trials. We have this issue of, on the one hand, patients want to be on trials, so they have to figure out how to get into the trial. And we have the pharmas that need patients to be on trials. And the problem is that match in the middle isn't so easy to pull off. So what did he do? You know, he built this workbench that provided a means of getting the accuracy of the match from 60% up to 90%. That's a 30 point gap that he addressed, and that's a much, much better structure. He drew the distinction between things that are kind of underappreciated in terms of ranking versus precision and the match of the trial. He's trying to give you sort of a top 10. He's trying to look at it and say, you know, Are you a good match for this trial? And if so, yes or no? And where are these matches? Where do they line up? And he gives a dashboard of sorts, which rates red or green, whether you're matching the criteria or not. And it gives a very, very specific way of doing these matches and gets the match quality much higher, which means when the patient is presented to the pharma company for the. For the clinical trial, the chances that they're going to be a match for that trial are much, much higher. I mean, right now, the statistic is something like somewhere between 5, 6% of patients who are theoretically a match actually get on the trial. He can raise that up dramatically because the matches are much more, you know, much better matched. And then there's a deeper insight here, which is this prompting workbench, which he can then use with the medical experts, who themselves may not be engineers, but they can encode their domain knowledge into structured prompts, which makes it even more specific for the protocols. And the knowledge accumulates of the scaffolding over time. And overall, the large language model helps the outcomes tremendously improve. They plug it in, and the scaffolding will work. And it gives the patients chances of getting on a trial because they're a better match, a much higher shot on goal.
- [14:05]
B
Yeah. What I like about Adam's story is that it directly challenges a dominant AI narrative, which is that the frontier model is what matters. That may have been the case in 2022 and 2023, but what Blum is saying is that the model now is a commodity. Right. It's good enough where you're getting frontier intelligence as a commodity for a price. The investment thesis should be build better scaffolding. How does that land Steve in an industry that's obsessed with model benchmarks?
- [14:43]
A
That's a great question. I think, like I said, pharma has been struggling with this problem for a really long time. And the fact that a lot of clinical trials fail, not because the drug doesn't work, not because the protocol is necessarily bad, because they can't find enough matching patients. And that problem has gotten substantially worse. In the age of personalized medicine, the criteria to get on a trial have gotten really restrictive. A clinical trial back maybe 15 years ago would have. There's a statistic we use in the industry, and it's called per patient per site per month. And 15 years ago, a difficult recruiting trial would have a PSM of roughly 0.5. Now those percent of those numbers are now not 0.5, but down like 0.01 or 0.001. And those with that level of specificity in the trial, any little thing could throw it off, making recruiting much more difficult. So a tool like Adams actually really helps with that problem of getting more specific specificity around the recruitment criteria and getting the patients identified for the trial. So it's sort of like this is a positive flywheel. As the system gets built, the protocols themselves can build this right in and it helps in a positive flywheel effect kind of way.
- [16:07]
B
Yeah, I mean, a really striking number, you know, there. I mean, it sounds like you're describing a situation where if you get one patient a year for a site, like, you're doing great, which sounds really cool in many cases.
- [16:20]
A
That's exactly right. And that's, that's, that's frightening because it cost somewhere between a quarter million, half a million dollars to set a site going, get a site moving, to stand it up. Absolutely.
- [16:30]
B
So with. From such a low baseline, a 50% improvement is, you know, is enormous. Right? Yes. Yeah, it's Bama. Yeah. So the, you know, Adam's just such an exceptional guy, but, you know, we could say a lot about him, but the patient is founder angle is interesting in the abstract. Right. So Adam didn't just build a tool for patients, he built one as a patient. Right. And he knew what the experience felt like from the inside.
- [16:57]
A
Right.
- [16:57]
B
And, you know, not every patient is a AI CEO. But is that a coincidence or does it tell us something about who should be designing healthcare AI?
- [17:07]
A
Well, I think what we're going to. I'm going to put a pin in that one, because in a few minutes we're going to be talking about Amy Price, who's talking about, in her segment in her podcast about how patients need to participate more in the construction of things. So let's put a pin in that question and come back to it in one or two minutes when we touch on Amy's. On Amy's area. So, you know, we have this issue of data quality as a foundation, and the scaffolding is the product layer, but there's this human piece. You know, when AI takes over a task, what happens to the people who use it to do it? Giovanni Donatelli from the language group, you know, maybe he's going to talk. He told us about the clearest answer to this that we've ever gotten.
- [17:50]
B
Yeah, let's talk about the evolving human role in AI. So Giovanni Donatelli founded the language group and built Fetch, a platform for AI assisted medical translation. And the core problem is deeply mundane. Right. When a patient with limited English proficiency gets discharged from a hospital. The instructions need to be translated accurately. Get it wrong and the patient comes back and not like, enough fun. I'm happy to, you know, happy to be back sort of way, right? The readmission rate for LEP patients, that's limited English proficiency is 14 to 18%. And one health system estimates the cost is about $4.6 million. So non trivial. What makes Donatelli's story distinctive is what he says about his translators. He doesn't say AI will replace him. He says, and this is a direct quote, there's a future for my translators. The future is AI handles the initial translation draft and a translator shift to training AI and quality assuring its output. The humans go from doing the work to ensuring the work is done right. And there's an immigrant story there that I really relate to. Right? And I was also a first generation immigrant is and it gives an emotional way. Donatelli's personal connection to language barriers is an abstract. He built this and got into the space because he understands what it feels like to be not to not be understood in a medical setting and to be misunderstood. And the mundane wins. Thesis isn't just about efficiency. There's a health equity dimension that's easy to miss, right? Like who gets the better care depending on who they are. So both of those jumped out at me.
- [19:27]
A
So the workforce evolution pattern, the translator becoming the AI trainer. It's exactly what we've been hearing about in other fields, like radiologists reviewing AI reads or radiology reads, coders reviewing AI generated coding systems, like we had with Josh. I always get his name wrong. Josh Gileras. And when humans are in the loop, it actually gives that extra component to it. Because neither of those firms are doing this without humans in the loop. They're making sure that the last step is that a human reviews. And Donatelli's company is no exception.
- [20:08]
B
I think there's a generalizable pattern here and I think that we both agree with that. I'll tell you guys a funny story from my past. I, I, when I was a graduate student studying cognitive science, you know, when people ask me what I wanted to do when I grow, you know, who I wanted to be when I grow up, right? And what kind of work I was envisioning doing. You know, my joke answer was, oh, you know, my dream scenario is I'm going to be a remedial tutor for dumb AIs. And this was really funny. In 1992, right? Like in, and, and in 1996. Wait, wait, go back.
- [20:41]
A
You were Using. You were talking about AIs in 1992 and 1996. Really?
- [20:46]
B
Oh, yeah, yeah, absolutely. Yeah. I mean, all right, you know, AI didn't just show up in 2022. People in cognitive science have been talking about AI since, what, 1956.
- [20:55]
A
Okay.
- [20:55]
B
It was the first. The first conference, so I wasn't that one. I'm not that old, but I was, you know, I was there in the 90s. You know, we used to do neural networks by the end, right? You just, like, you just compute the weights and say, oh, yeah, you know, got the Excel spreadsheet on it. So, you know, but. But that was the. That was a funny joke. And it stopped being a joke in the 2000s. Right? Now, that is what I do. I'm just a remedial tutor to dumb AIs. I. I tell them, you know, what they're doing wrong, or at least what they're doing that I don't like, because I'm not. I'm not sure who's tutoring who at this point. So there is a work workforce evolution story there that could be told as like, well, we are all becoming AI coaches by profession, and some of us are going to be AI validators and some are going to be AI watchdogs. But it doesn't seem to be at least a lot of fields, a displacement story. Although I don't want to be overly optimistic. I think there are many tasks and perhaps some jobs that will be automated away just like they were when telephone operators got displaced by PB access. But I think it's more interesting to consider where will human attention need to shift when we can automate the stuff that can be automated with emerging technologies?
- [22:11]
A
Well, that whole thread reminds me about when we listened to Cathy Rowe and Kenny white in episode 17 when they were talking about the malpractice gap. Who actually becomes responsible when the AI gets it wrong? In these cases, we have humans in the loop, but that may not be the case for many more years. You know, in five years, it may not have humans in the loop in quite the same way. The responsibility for the legal. And you're the lawyer among us, right? So I'm the doctor, you're the lawyer. Where does that take us?
- [22:43]
B
I think what Kenny and Kathy reminded us of is laws don't go away just because technology emerges. You know, we have several models of liability. There's malpractice liability. You know, offered a professional service and you didn't do it up to professional standards. There's product liability. You built something and it didn't. It hurts the way you built, it was faulty and it hurts somebody. And there's sort of general liability principles which is you committed or you, you, you committed a wrong. Right. You were negligent or it was intentional. You could, you had an intentional harm. All of those concepts in the law still exist and they exist whether or not we have AI. So that's, I, I think that's a really useful reminder as to where the liability falls. Depending on, in an environment where you have rapidly evolving technology, it's not clear. Usually liability doesn't go away. So right now the liability just falls on the dock. Right. So if some, if a patient gets hurt and the doctor has malpractice insurance, I think the standard assumption is, yeah, it's your problem. There are going to be arguments that will be made that you know about when that liability shifts over to the technology manufacturer. Right. So if a radiologist is using a CT scan and the CT machine malfunctions. Right. The radiologist only liable for, for not performing their, their portion of professional services correctly. The Siemens or whoever built the CT machine is liable for the machine breaking. So some of that is going to translate pretty well. I, you know, in reality with how those boundaries would be drawn, I think that's a very rapidly moving target and boy, do I not have any intuition about it. But, you know, I think we just need to bring, bring folks like Kenny and Kathy back to see how they're thinking about the problem and what they're seeing because the courts will have to decide.
- [24:33]
A
So, you know, discharge translation feels like a pretty mundane use case. And you know, we did talk about mundane use cases in the first block as being those that really matter. But you know, in this case, they're seeing real dollars out of this, this mundane use case. Four and a half million dollars, 4.6 million DOL. You know, to drop readmission rates. That's serious money. And I guess the purist might say mundane wins. Examples that we've had, you know, and even more so than Josh Gileras. What do you think?
- [25:06]
B
I love that we've now the technology has moved forward enough where we think like automatically translating from one language to another is mundane.
- [25:13]
A
Yeah, that's very true.
- [25:14]
B
We're like, oh, that's nothing. You know, just take, just take one language and translate. But yeah, I think I agree it's a really nice, very focused example of like a specific function where the automation, you know, what is to be automated is pretty clear and the cost savings and time savings are pretty clear. So I think it sort of confirms our thesis. That's a very good, those are really good places to start the automation journey. So we talked about the data foundation so far, the structure around AI and the evolving human world. There's one more infrastructure layer. Who owns the data in the first place? And Shashi Shankar had a really provocative answer for us. Which brings us to our theme of patient data ownership.
- [25:57]
A
Yeah, and that's been a core point in my career. Who owns the data? Different places in the scheme have asserted ownership various times, but at the end of the day it's pretty much the patient owns the data. Shashi came from Genentech and he now runs Novelia, which is building something that didn't exist as a category even five years ago. It's patient authorized real world data. The premise is pretty straightforward. Today if a biopharma or anybody who needs to use real world data needs to get it, they'll go to IQV or Flatiron or some other health information exchange and they'll make a transaction to obtain a lease or in some other way gain access to using these data sets. Well, those data sets typically can be all incomplete, but in various ways claim data that doesn't have any detail clinical data from a health system without any of the others in the system. Shashi has an argument which is basically the only person who has the complete medical record, bar none, is the patient. The patient knows their life. The patient can integrate all this. And that's his sort of thesis. And that perspective was difficult to really leverage until recently when the 21st Century Cures act came out and said in effect that patients now have the legal right to consolidate their own data. You know, that's one of the places where Health Vault and Google Health kind of fell down back in the early or the mid 2000 2010s. You know, the regulatory foundation literally didn't exist. Patients could not compel their providers to share. And even some today, even though it's laws still have some difficulty with it, but now they can. There's a law that specifically says that. So you know, he looks at this issue and he calls it the skinny tail. He actually did something also radically different. He didn't go after, you know, the worried. Well, he went after the sickest, most complicated patients to try to build their medical record, real world evidence, data sets from that part of the healthcare ecosystem because that's where there's a lot of information. It's really important information. The cases are complicated and he's leveraging all of the above to come up with very sophisticated real world evidence data sets. And you know, the question is, are the big tech organizations, are they really the organizations we should be trusting with all of our sacred data or is it the patient? What do you think?
- [28:18]
B
You know, we, we pushed Shankar pretty hard on the failure record in this industry. So I think you and I have both been really interested in patient data ownership. Right. But a lot of the work in registries, at least in the last decade, has been relying on patient being the aggregator of data. And overall there's just a really compelling ethical angle to that model. But it's always been really difficult. Right. You know, there's a, the, the, the graveyard of companies that have tried to do patient data banks is really full. Right. So, but there's something about what there he's doing and so, and the timing about what that, about what they're doing is really compelling. It does require patients to actively participate and to consolidate their records and to authorize access. Now that works for motivated tech savvy patients like some of our friends like you know, Epatient Dave, Dave Brunkhardt, Bronkhart and Amy Price, who we'll talk about in a little bit. But what about the normies, right. You know, the population that needs it most, like elderly patients, those with limited digital literacy. Is there a health equity gap in patient authorized data?
- [29:32]
A
You know, I think it's almost intrinsic to the space that there probably is. But I think that the more we continue down this road, you know, perhaps it'll, that gap will close. You know, I, I have a friend who's working on a, on an AI ambient listening tool. Maybe we'll bring one as a guest at some point. And the challenge there is that he's working with elderly, working on the, the doctor patient interaction, trying to bring that to bear. But one of the biggest problems that he's facing is that elderly folks are just not that it. Literate for the most part. I mean there's obviously exceptions and the engaged patients like you and me, we're
- [30:14]
B
still hanging in there.
- [30:14]
A
We're still hanging in there. But we're only. I'm in my early 60s. You're in your late 50s. We're not the ones I'm talking about. We're talking about like if my grandparents are still alive, that's who I'd be talking about. People.
- [30:25]
B
Oh no, my kids definitely think I'm elderly. So you know, we're just.
- [30:28]
A
Yeah, but I think the point is that figuring out a way to help bridge that gap with, with you called it one Thing. I call it the worried well, but I think that gap is going to have to get crossed. I don't know that it's going to be so easy just yet, but maybe there's another use case for AI to help with that piece of it as well. So long as it's sufficiently simple that it can do it without adding too much cognitive load to like a 70 year old or 75 year old patient who's not tech savvy. I don't know.
- [30:58]
B
That's super interesting that actually, you know. So AI is the bridge for data collection as sort of an equalizer for the patients who are struggling to even get to the basics of authorizing the data that actually connects to. You know, another point that was really important with Shashi Shankar is he's using LLMs for the data curation pipeline. Right. He's using for normalization to duplication coding across disparate record formats that used to be the valley of death with patient derived data integration. I mean, it was just brutal work to get those. Not just get the documents from the hospitals, but then convert them into something remotely useful. It's an interesting inversion. You're using AI to prepare data for other AI systems. Is the data infrastructure itself becoming an AI driven application?
- [31:49]
A
Okay, that seems like a recursive question, which I don't know that I'm prepared to answer at the moment because I think you're right. I think it's, you know, the AI is going to be helping the AI and you know, at some point, you know, and there'll be hopefully humans continuing to be in the loop. But it really.
- [32:06]
B
AI turtles all the way down, Steve, it's a what? It's AI Turtles all the way down.
- [32:11]
A
All the way down. Okay.
- [32:12]
B
Oh, you, you know, you know the joke about it's turtles all the way down?
- [32:15]
A
No, I don't.
- [32:15]
B
It's. Oh, it's, it's an old philosophy of science joke. So, you know, a elderly flat earther is a physics conference and he says, you know, I, I know, I know you're what y', all, you all are describing, but you're all wrong. You know, you know, the earth is standing on the back of a turtle. The earth rests on the back of a turtle. And of course, the physicist at the podium says, ah, but what does the turtle says on. And the old flat earther laughs and says, you can't fool me, sonny. It's turtles all the way down.
- [32:44]
A
I'd actually never heard that before, but okay, it's classic.
- [32:48]
B
I'm glad we're sharing it with the audience. Yeah, so, yeah, so it's. It's AI turtles all the way down in this recursive world.
- [32:55]
A
So, you know, we have patients who are owning their own data, and the data is, you know, one dimension. But we're going to go back to that pin I put up a couple minutes ago about Amy, Amy Price. And she takes it in another direction where patients aren't just the data sources, they're the co designers. And her perspectives tie kind of everything we're talking about together. At least I think they do. What are your thoughts?
- [33:21]
B
So Amy is a good friend of ours. I'm just really fond of her. So, you know, but she happens to be the editor in chief of the journal Participatory Medicine and a researcher at Oxford and Dartmouth. And she brought a frame that I think ties this entire block together. And I think she said some really powerful things. So one thing she said that really redefined the human in a loop concept, or human on the loop, slightly different concept is she really pushed back. She's like, it's not a human in the loop. It's a knowledgeable human who cares. And the distinction matters. A human in the loop is a checkbox. A knowledgeable human who cares is a partner. So she describes participatory medicine as a symphony. You need to practice together, develop relationships and learn rhythms. It's not just a checklist exercise. It's not a technology problem. It's an organizational design problem. It's a human problem. And she pointed to something called a preference erasure, which is that systems record patient preference sometimes, but never surface them at the point of care. So you ask the patient, how would you prefer this to be done? And that gets recorded, but it never appears. And the data exists, but the workflow doesn't use it. But the line that I think captures this entire block for me, and I just want to give Amy credit for this, is that she pushed back on. I asked her why this system is broken. I was using Break it Cyber. She said, I just don't agree that it's broken. A broken system means you throw it out. An unfinished system means that there are errors along the way, and it's just part of the process. So healthcare AI isn't broken, it's unfinished. And I just love that distinction. Like, it completely escaped me that how important it is to differentiate and how difficult it is to differentiate systems that aren't working because they're broken from the ones that aren't working because you're still building them. You can't move into the building yet, but it's not broken. They're just pulling up drywall, man. So the frameworks exist in healthcare. We have Cochrane reviews, we have reporting standards, we have fire now, we have section 1557, we have the 21st Century Cures Act. They just haven't been connected yet. And I think that's the story of Block 4. We're watching the connections being built and the wiring starting to come together.
- [35:40]
A
You know, it's like, you know, in some respects, it reminds me of the old EDS commercial, which is building the plane in flight. I don't know if you've ever seen that little video, but for those of you who haven't go on YouTube, query EDS, the building the plane in flight, and you'll see what I'm talking about, because it feels a lot like that, like we're building everything as we're going and things are getting better and better, but, you know, you drop a wrench, it's going to go to the ground as we're doing it. You know, Amy said something else, that, which, which I want to come back to, and you flagged it, which is that anybody can become a knowledgeable human given the right tools. And that's like an empowerment argument, you know, AI literacy as. As a patient. Right. But is healthcare as a system, is it ready for that? I mean, sometimes we work around the patients, in fact, and not necessarily intentionally, but because there's only so much time in a day, doctors are overburdened. But, you know, as we're building these tools out, maybe there's a different path.
- [36:44]
B
I don't know. I struggle with the question of how do we help people get started with AI? So when I talk to colleagues about it, you know, somebody who uses really proficient in AI tools, when people ask me, like, what do you do? What did you do to get started? Right. I tell them, you have to get started. Right. It's like, you know, you can't learn to play the piano by. By watching people play the piano or by reading, by playing the piano. You kind of sit down and start practicing. But I don't know how well that works with patients. I think that that's a good argument for sort of data scientists and informaticists and clinicians, but I really don't see the bridge yet. Right. I know we will figure something out. So I'm optimistic about our abilities to solve problems, but I am trying to be honest about not knowing the solution. I just, to me, this is a huge gap. And like, you've been writing a papers on AI literacy. Right. Did you guys come up with anything? Well, you know that.
- [37:47]
A
Yeah, we. On the literature and the paper was just submitted last Friday, knock on wood. It gets accepted. But the folks that worked on it at the DCI network, we were really happy this one got out the door in under a year. It got out the door in a little over five months, which was a phenomenal thing given how much work it is to get an academic paper out the door. But in AI literacy, it's a continuum and we're working on building out this. This. We're going to be doing a survey on AI literacy to sort of identify what are the points that make up this gray. This gray space in between, you know, completely AI illiterate to being an AI literate individual or company, to sort of identify where those gaps are and how to try to bridge those gaps. So, you know, that's kind of. This issue of AI literacy, I think, is one of the least spoken about issues, but it's actually one of the most important issues that we. That's why we wrote the paper, because all of this came out during the course of our. Of our Signal through the Noise conference last September. So let's. We'll put a pin in that until the paper comes out and we see how it's received, whether it gets accepted or not. But I think AI literacy is a critical piece that people are just not paying a ton of attention to. You know, Amy pushed back on something else. She called out that hallucinations are basically judgmental language. She called it, you know, hallucination is an AI error. Instead she framed it as. It's just more like a bright junior associate and it. Who's someone who needs to be, you know, supervised. Is that the right metaphor for where we are? Is AI? Not AI is a threat, but AI is a savior, or is it a junior colleague that we need to teach and train and help improve?
- [39:41]
B
Look, I really like that framing. I wasn't super sensitive like Amy was about the negative connotations of hallucination because like, I describe myself as hallucinating all the time. I was in a meeting yesterday where I said, did I hallucinate that? Because I couldn't remember if we agreed to something or not. And, you know, so like Leon, AI hallucinated the fact that this, you know, the consensus was achieved. Right. So. But she is right in one sense. I think, describing having this kind of overly psychological and overly negative framing on a functioning of a system that just needs to be engineered around is probably not helpful. And I know we, today we have engineering approaches that could dramatically reduce the kinds of errors that people were experiencing. Were they using AI as a chatbot, you know, without, without any real guardrails or thoughtful frameworks. And we see that in tools like Open Evidence, for example, that we've had on earlier, we discussed earlier in the podcast. I think it's, I'm reminded of something I heard recently from Terence Tao, who is probably one of the most eminent mathematicians of the modern age. And he's, I think a few years ago he talked about AI as sort of being like, oh, interesting and occasionally helpful, but not terribly useful. And he gave a talk recently where he described that, you know, he just works with AI regularly and now thinks of it as a junior colleague. I mean, it raises questions in my mind. Is there going to be a point where he starts thinking of AI as a senior colleague? Right. You know, and I'm not fear mongering, I'm just sort of, it's a natural progression. And you kind of say like, yeah, you know, went from like kind of an intern to, you know, junior to associate professor, you know, to full professor to write something else. And it's a, it's an interesting progression to think about. But certainly today it may be really good framing right in, in 2026 with the Frontier models and a good engineering approach, you can really start thinking about AI as a junior colleague that sort of, that's a little bit off, maybe a little bit psychologically different from you and you need to manage them carefully. But I think that's a really helpful frame.
- [41:59]
A
You know, just taking a step back and looking at it from the top down. In 2022, these models didn't exist until like November. And we've gone from non existence to, you know, getting 70 or so percent on the, on the medical boards to now getting 95 plus percent on the medical boards. Junior colleague. Yeah, I think it's going to be more than a junior colleague relatively soon, honestly. But let's continue down this road because we're wrapping up towards the end of this discussion. If we zoom out across all five episodes in this block, we heard Charlie Harp talk about data quality, Adam Bloom talking about the scaffolding, Gio Donatelli talking about workforce evolution, Sashi talking about patient data ownership, and now Amy's perspectives on participatory medicine. They're all saying kind of the same thing, that the technology is ready and that the infrastructure really isn't quite yet is that maturation story.
- [42:59]
B
For me, it's, it Captures something important.
- [43:03]
A
Right.
- [43:03]
B
It's. It's like when, you know, during this, during rapid technology progressions, you can have an interesting phase where the internal combustion engine is giving you unprecedented power efficiency ratio, you know, power to weight ratios and. But you haven't figured out, like, the brakes in the steering yet, you know, and the windshield wipers are not working right. Because nobody knows, nobody's realized you need windshield wipers.
- [43:27]
A
Windshields. Yes.
- [43:28]
B
Yeah, yeah. There's like, why are there bugs in my face now? So, so you have. So I think we're at that point where they're part of the system that have just jumped ahead and people are playing around to see what else do you need to build around it in order to have a working car. Right. Because we know we need to perform human functions that meet human needs, like getting from place to place safely. And I think it's a little bit harder when you're talking about a general purpose tool like AI, But I'm hoping that metaphor will hold and we'll be able to break it down. So I think those are five themes. There's a data foundation, a scaffolding principle, a workforce evolution, a data ownership shift, and a design philosophy. And the thread connecting all of them is that the technology isn't the bottleneck anymore. So let's talk about what that means from where we've been and where we're going. Here's what I find remarkable. In block one, we were still defining our terms. We're like, what is this thing? What are mundane wins? What does it mean to have governance versus regulation? In block two, we're checking those terms against history. In block three, we're testing them against evidence. And now block four, nobody's debating the terms anymore and the conversation is really moved into. And this has happened in like, less than a year, right? We've been at it less than a year, six months.
- [44:44]
A
We've been only at this, so six months.
- [44:46]
B
And we're, you know, so now data quality, scaffolding, human roles, patient agency. That's what that is, what maturation looks like. It's not a. Again, it's not drama. It's, you know, it's not. It's not a Eureka moment. It's a quiet shift in what the conversation is about. So I think we're trying to shed some light on that shift.
- [45:05]
A
So let's build on that. You know, what strikes me about this block is that every guest was a builder. Like, if you take a step back, each one of them was talking about building something. Charlie was building this data quality measurement instrument, the Picky framework. Adam was building the scaffolding to help improve clinical trial matching for patients. Gio Dantelli was talking about this translation infrastructure for physical language translation. Shashi was talking about building the data for the patient's data pipes. And Amy was advocating for participatory design. None of them were debating whether the AI was working. And again, this is like episode what, 30, 31? Something like that. That's not a long time for this conversation to really shift to this degree. It reminds me, just a few weeks ago, I was at the HIMSS conference and I noticed a tangible difference of what I was seeing walking around the trade floor and going to some of the lectures, which was that I saw a bunch of companies there that had been there a couple years back. And while they were touting, look what I can do back then, now they're touting, look what I'm doing and how it's having a return on investment and how it's actually moving the needle in a very real and positive way that, in fact, investors care about that patients care about that clinicians care about. And that was completely absent the last time I attended himss. And it was a tangible difference in the atmosphere of the conference and how people were talking at their booths and so forth and so on. So, you know, I'm not sure where to go with that, but it feels to me like in just a really short period of time, things have pivoted pretty dramatically. I don't know if it's us because we're how we're focusing here, but we didn't have these conversations when we started this podcast.
- [46:52]
B
I think we were having a hard time finding success stories when we started.
- [46:56]
A
Yeah, right.
- [46:56]
B
I remember, like, we were, you know, a year ago. What we're like, who is doing this? Well, right. And I think I sort of agree that the industry maturation markers of vivid, if you know what to look for, right? If you're starting to look for, like, you know, we need AI became like, what data do we need and eliminate? The human became restructured. The human's role and regulation is a constraint and is annoying. Became regulations, creates the conditions that we need for growth. And these are, you know, these are phase shifts, right? Like, this is. These are cat. You know, they are changing the way the field is thinking about AI, how.
- [47:34]
A
But I mean, look, you and I both live through the Internet. The Internet bubble. When it happened, I don't remember it happening this fast. I mean, Internet came online the way we know it today with the web roughly 95ish. But like Amazon wasn't a real big thing until maybe 2003, 2004, and it was questionable whether it would even survive. You know, ebay wasn't really making anything. But my point is it took seven years, six, seven years before we started seeing use cases that were really making a difference. This has changed in less than three. It's half the time. And the pivot that I saw at HIMS is tangible in that same period, in that three year time span. So it really feels like we're moving at a whole different pace. So where does that leave us? We've done through like 30 episodes. We've had four reflection blocks. How do we update our thesis? And what do we think? It's still missing.
- [48:34]
B
So I think our thesis has become more layered, right? So we, we basically have block one was mundane wins matter. Block two was learn from the EHR era mistakes. Block three is governance, economics, institutions and epistemology, all constrained success. And block four is infrastructure layer, data scaffolding, human roles, patient agency is where the real work is going to happen. And the next question I think is what happens when the infrastructure meets the institution. And we got a glimpse of that with Ted Shortlev and Matt Troupo. These are episodes we'll save for next time. We just recorded them and they will be coming out after this reflections episode. So just to bring it together, block four gave us the infrastructure layer. HARP showed us that the data quality is the foundation AI needs and that AI is the first system that actually demands that level of quality. And Blum showed us that the scaffolding around AI matter more than the model itself in many cases. Then Atelier gave us the clearest workforce evolution template we've had. Shankar showed us that the patient data ownership is a legal right waiting for the right infrastructure. And Price gave us the design philosophy, which is going to be my new motto for AI in healthcare, which is unfinished, not broken. The thesis we've been building since episode two now has these four layers, mundane ones, matter, history, rhymes, evidence, tests and constraints. And infrastructure is where the real work happens. That's, that's where I think we're at after episode 30. And we've only begun to fight.
- [50:09]
A
Yes, indeed. So, you know, we want to say, you know, hopefully this was helpful to our listener base. It takes us a while to get through these, the construction of these reflection episodes. And I don't know if you'll agree with us, if it resonates with you or not. This is our thesis of our thinking. We hope that when we bring on future guests like Ted Shortliff and Matt Troupo from Sanofi that those things will carry through in those conversations. And then what happens is decades of AI history all meet today's infrastructure builders. We'll see where that goes.
- [50:44]
B
And with that, I'd like to thank our amazing guests from both Block 4 and the previous blocks who made all of these reflections possible and gave us so much to reflect on. I want to thank the audience for joining us and if something in today's conversation resonated, send us a comment and share it with a colleague on. Join us on LinkedIn and follow our newsletter and please join us next time on Practical AI in Healthcare.
- [51:13]
A
Thank you for joining us this week on Practical AI in Healthcare. If you're ready to go beyond buzzwords and hype and explore how AI is truly transforming healthcare, stay tuned for more conversations that get us to what works. Until next time, stay practical.