S1, E44 - Reflections #6: After Five Episodes, What Still Has to Be Human?

Summary

Practical AI in Healthcare – S1, E44: Reflections #6 — After Five Episodes, What Still Has to Be Human?

Hosts: Dr. Steven Lapkoff & Dr. Leon Rozenblitt
Release Date: July 5, 2026

---

Episode Overview

In this sixth “Reflections” installment, hosts Steven Lapkoff and Leon Rozenblitt look back over the previous five episodes to synthesize key themes, lessons, and persistent questions. Central to their discussion: When it comes to AI in healthcare, where does the human skill, insight, and judgment remain irreplaceable? Through a lively breakdown of five interviews—with Sarah Rossetti, Jeff Smith, Hugo Campos, Fred Bennett, and Zach Kahani—they highlight breakthroughs in practical AI, examine both progress and pitfalls, and repeatedly return to the enduring value of human ingenuity, caution, and context.

---

Key Discussion Points and Insights

Sarah Rossetti: The Human Signal Behind the Data

Guest: Dr. Sarah Rossetti, Asst. Prof. of Biomedical Informatics & Nursing, Columbia; PI of the Concern Project

• Main Contribution:
  Rossetti’s team discovered that the “density and frequency of nursing documentation” can itself serve as a vital, predictive signal—often indicating heightened concern from nurses before traditional vital signs flag deterioration.

  “The fact that a nurse goes to take vitals at 3am and wakes the patient up tells you that the nurse is worried about the patient.”
  – Dr. Leanne Rozenblitt (02:40)

• Observational Research Metaphor:
  Rozenblitt likens using real-world data to “picking up whatever the tide left on the beach”—most is driftwood and seaweed, but creativity is finding or repurposing the ‘treasure’ that’s present.

  “There’s a lot of cleverness in hunting through the seaweed and driftwood and picking out the bits of treasure…Another kind is figuring how to use the driftwood—to build a hut or turn the seaweed into fertilizer.”
  – Dr. Leanne Rozenblitt (05:27)

• Human Creativity vs. AI Automation:
  The hosts stress that it was a human who recognized the signal and designed the approach—while AI made large-scale detection possible, this was not an “AI-generated idea,” but “human creativity enabled by AI.”

  “This insight… wasn’t an AI generated thought… For those out there who are concerned that AI is going to replace us all, I think this is a great example where a human idea, a human observation is something that became... the survivability... has been proven”
  – Dr. Steven Lapkoff (09:43)

• Minimal Disruption:
  The elegance of Rossetti’s method: no extra data collection is needed. It leverages “digital exhaust” already present in EHR systems.

• Nuance and Context Pitfalls:
  Both emphasize the need for clinical interpretation:

  “Unless you understand the data from a clinical perspective, you might treat the data as gospel truth.”
  – Dr. Steven Lapkoff (07:47)

---

Jeff Smith: Agents, Access, and Unintended Consequences

Guest: Jeff Smith (US government, ONC)

• Data Access Rights for Agents:
  Smith explained ONC’s move toward granting authorized ‘agents’ (software or organizations acting for a patient) the same data rights as patients themselves, to prevent data blocking.

  “If you don’t let an agent have access to the data with appropriate consents… that could be a fineable offense under… ONC.”
  – Dr. Steven Lapkoff (12:55)

• Chain of Delegation and Consent:
  The hosts highlight slippery slopes:

  “What happens if a patient authorizes an agent… then that right gets passed to a third party…the company goes out of business and the rights get inherited by the creditor? Eventually, some party I never interacted with can get that right.”
  – Dr. Leanne Rozenblitt (15:15)

• Reference to Sci-Fi and Smart Contracts:
  Rozenblitt draws a parallel to cyberpunk sci-fi—intelligent, autonomous contracts negotiating rights without humans.

• Potential for Unforeseen Consequences:
  Both agree: significant, unpredictable downstream effects are likely.

• Upcoming Changes to Data Writing Rights:
  Smith is considering rules that would prevent data blocking not just for access, but for writing to healthcare records—potentially game-changing, but with unclear ramifications.

---

Hugo Campos: Empowerment—But at a Cost?

Guest: Hugo Campos (patient advocate & technologist)

• AI-Enabled Patient Self-Advocacy:
  Campos uses AI to access and interpret his own medical data, but admits he doesn’t always fully understand the output.

  “He doesn’t quite understand what he’s built. And that’s concerning… as a builder, that worries me.”
  – Dr. Leanne Rozenblitt (18:35)

• Black Box Dilemma & AI Literacy:
  The risk: Empowering patients via AI can backfire if they lack clinical context—leading to erroneous conclusions about their health.

  “She made a diagnosis of herself recently where it was completely 180 degrees wrong… because she didn’t have the context to interpret the data.”
  – Dr. Steven Lapkoff (21:04)

• Strengths in Open Source & Systems Thinking:
  Campos’s compensatory strategies:

  • Cross-validates output with consensus from multiple AIs—mirroring best practices in AI development.
  • Publishes open-source tools to allow for greater transparency and collaboration.

• AI’s Barrier-Lowering Power:
  LLMs are described as “the first software tool…that didn’t make me feel stupid.”

  “Tools for a novice user are really disempowering… but LLM models… using natural language as the interface, they’re very tolerant of human error.”
  – Dr. Leanne Rozenblitt (24:57)

  Lapkoff shares his own story:

  “I built the same damn [sentiment analysis] pipeline in an afternoon, like two and half, three hours, which two years ago cost $100,000.” (26:56)

---

Fred Bennett: The Rashomon Problem and "Minimal Trustable Product"

Guest: Fred Bennett (Founder, Patient Talker)

• Ambient Listening for Patients:
  Instead of documenting for providers (Abridge, Nabla, etc.), Patient Talker records doctor-patient encounters to generate a digest aimed at helping patients really understand their care plans—solving the “Rashomon” problem where everyone remembers a different version.

  “Fred’s mom and his dad were all listening to the same 30-minute encounter with a doc and came away with dramatically different perspectives on what was said.”
  – Dr. Steven Lapkoff (29:44)

• Communication is What the Listener Does:
  Rozenblitt highlights a shift:

  “Communication is what the listener does… it didn’t happen just because you said something.” (33:12)

• Trust vs. Minimal Product:
  In healthcare, “minimum viable product” isn’t enough; you need a “minimal trustable product”—something that earns the user’s confidence and safety.

  “Trust is just fundamental. If it’s not trustworthy in healthcare, it is not viable.”
  – Dr. Leanne Rozenblitt (33:22)

• Business Model Challenges:
  Discussion of whether stand-alone patient AI tools are sustainable or ought to be part of a broader chronic care or care navigation offering (including insurance billing alignment, codes, etc.).

---

Zach Kahani: What Values Do Our AI Models Hold?

Guest: Dr. Zach Kahani (AI & healthcare informatics pioneer)

• Landmark Question:
  Kahani’s signature move: “What values do our AI models express?” We don’t know what values drive model recommendations unless we probe their choices, not their stated rules.

  “These systems are behaving as if they had values…their behavior represents something…You can measure what their values are by giving them choices and seeing how they go—boom. Mind-blowing.”
  – Dr. Leanne Rozenblitt (39:38)

• Analogy: App Ecosystems and Alignment Problem:
  Kahani was among the first to ask why EHRs can’t be open “app platforms” like iPhones (inspiring SMART on FHIR). Now, his ethical focus: what happens when AIs increasingly make decisions?

• Data Sources Encode Biases By Default:
  Hosts remind that EHR and real-world data—training material for AI—are “already baked” with historical, unspoken human values and biases.

  “It’s like trying to take the sugar out of a cookie.”
  – Dr. Steven Lapkoff (41:49)

• Trolley Problem for Healthcare AI:
  Explicitly defining clinical “values” (e.g., whose outcome gets prioritized) is even harder than in self-driving cars because so much is hidden, unconscious, and emotionally fraught.

  “People are hypocritical. We do things that we say we probably shouldn’t, all the time…making rules explicit is often very uncomfortable because it reveals the hypocrisy of both ourselves… and the organizations we serve.”
  – Dr. Leanne Rozenblitt (42:01)

• Attack Vectors & Model Poisoning:
  Kahani also points out the looming threat of “model poisoning”—maliciously flooding AIs with misleading info so their outputs shift subtly but dangerously.

  “If you send enough information to an AI model and it’s reading the information as a signal, and that signal is being repeated in a disproportionate way, that signal may get picked up as ground truth.”
  – Dr. Steven Lapkoff (44:46)

• The Answer: Model Monitoring and Auditing:
  Continuous “measuring the model’s behavior” is seen as the only way forward.

  “Ultimately, we have to measure the model’s behavior. And I think that’s the answer.”
  – Dr. Leanne Rozenblitt (46:17)

---

Notable Quotes & Memorable Moments

• On Human Creativity vs. AI Automation:
  “I love the fact that this was a human generated idea.”
  – Dr. Steven Lapkoff (09:43)

• On data-driven error:
  “The easiest person to fool is yourself.”
  – Dr. Leanne Rozenblitt (08:24)

• On regulatory risk:
  “With great power comes great responsibility.”
  – Dr. Steven Lapkoff (15:56)

• On patient tech empowerment:
  “It’s the first software tool…that didn’t make me feel stupid.”
  – Dr. Leanne Rozenblitt quoting her psychologist friend (24:57)

• On real-world AI ‘cracks’:
  “This AI stuff is like crack for nerds, man. I’m living that dream.”
  – Dr. Steven Lapkoff (27:33)

• On ‘minimal trustable product’:
  “If it’s not trustworthy in healthcare, it is not viable.”
  – Dr. Leanne Rozenblitt (33:23)

• On defining values in AI:
  “You can’t ask a person what their values are… but you can measure what their values are by giving them choices and seeing how they go.”
  – Dr. Leanne Rozenblitt (39:38)

• On model safety:
  “Who shall watch the watchers? It’s us, right? We apparently volunteered.”
  – Dr. Leanne Rozenblitt (47:44)

---

Timestamps for Segment Highlights

• Sarah Rossetti/Concern Project: 02:40 – 10:49
• Reflections on Observational Research: 05:27 – 07:47
• ONC/Jeff Smith & Regulatory Shifts: 12:55 – 18:22
• Data Inheritance/Chain of Delegation: 15:15 – 17:41
• Hugo Campos/Patient Empowerment: 18:35 – 25:58
• Patient Talker/Fred Bennett & Rashomon: 29:44 – 35:57
• “Minimal Trustable Product” in healthcare: 33:11 – 33:53
• Zach Kahani: Values Embedded in AI Models: 37:11 – 44:46
• Model Poisoning & AI Optimization Risks: 44:46 – 47:27

---

Conclusion: What Still Has to Be Human?

Throughout the episode, a central theme emerges: no matter how advanced AI becomes, critical elements—the insight to recognize new signals, the ability to contextualize data, the ethical judgment of values and tradeoffs, and the humility to question and audit our tools—remain stubbornly, beautifully human. The hosts urge continuous vigilance (“Who shall watch the watchers? It’s us.”), humility before complexity, and a commitment to building AI that amplifies—not erases—our finest human skills.

---

For more details, links to each guest, and the full episode library:
www.practicalaiinhealthcare.com/episodes

Transcript

[Narrator]

Welcome to Practical AI in Healthcare, the podcast that cuts through the noise to spotlight real world solutions delivering real world value. From patient care to clinical research, from life sciences to patient engagement, we focus on what truly matters in healthcare today. No hype, no theory, just practical insights where AI is making a true impact. Dr. Steven Lapkoff and Dr. Leanne Rosenblitt are your hosts as we explore what's real and moving the needle in this exciting new domain. Welcome aboard and let's get to it.

[Dr. Steven Lapkoff]

Hello and welcome to this week's edition of Practical AI in Healthcare. My name is Dr. Steven Lapkoff and I'm here with my colleague, Dr. Leon Rosenblit. How's it going, Leon?

[Dr. Leanne Rosenblitt]

Oh, much better, Steve. I'm recovering from my neck injury. I'm starting to feel human again.

[Dr. Steven Lapkoff]

Well, that's great. I know that it's been a big problem for the last few weeks for you and you missed a bunch of stuff and you couldn't travel and I'm glad to hear you're doing better. We have our conference this coming week up in Boston, the patient conference for.

[Dr. Leanne Rosenblitt]

I've been boosting myself up for that. I really don't want to miss that one. So like, I tell the physical therapist to beat me up extra hard so I can make it to Boston, so.

[Dr. Steven Lapkoff]

Well, I'm glad, I'm glad we'd get to get be able to get together in person and not over zoom for a change. So Leon, you know, it's time for us to do our next Reflections episode. Woo hoo.

[Dr. Leanne Rosenblitt]

Love those.

[Dr. Steven Lapkoff]

And for those of you in the audience who have been following practically at Healthcare every, you know, call it six to seven episodes, we try to go back through what we've heard and we try to synthesize some of the ideas, some of the themes, some of the things we've been hearing about in our episodes to try to connect some dots for the listeners. It also makes a bit of a cheat sheet for those of you who don't want to go back and listen to everything one off. But we encourage you to do that because the depth and detail of all these conversations is found in the original episodes. And for those of you who care, it's at www.practicalaiinhealthcare.com episodes. And that's where you can find all of our library and archive and it's all there for you. So Leon, why don't we dig right in and we had a bunch of folks that we're going to cover today. We're going to cover Sarah Rossetti, this episode Jeff Smith, Hugo Campos, Fred Bennett, and we'll top it all off with Zach Kahani at the end. So why don't we just dig in and talk about the kinds of things we heard about from Sarah Rossetti? As you'll recall, she's an assistant professor of biomedical informatics and nursing at Columbia, and she's the PI of the Concern Project, which was quite an interesting program.

[Dr. Leanne Rosenblitt]

Yeah, she was phenomenal. I think you and I were both really impressed by how clever her method was. And it was clever in, like, a quiet way, but a really, really deep way. So I think what. So for those listeners who didn't hear the episode, which I highly recommend, her team realized that the density and frequency of the nursing notes was the data. Right? So there's. Yes, there's signal in watching the patients. Yes, there's signal in what the nurses say. But the fact that a nurse goes to take vitals at 3am and wakes the patient up tells you that the nurse is worried about the patient. So just the fact that there's density of information and stuff is being written down at weird times is telling you that the nurse is worried, is telling you the level of concern, hence the clever acronym that the nurse has. And I think this is just a great reminder of what really thoughtful and creative observational research is about. You know, I don't know, and I'm happy to talk about that more. But, Steve, what did you think? What is your.

[Dr. Steven Lapkoff]

I agree with you. I think the fact that she found signal in. In what's otherwise relatively noisy environment, you know, ICU care. You know, I've spent many, many nights and days in ICUs and CCUs, and they are among the noisiest places, ironically, in the entire hospital. And they're not noisy necessarily with voices and with. With sounds, although there's a lot of alarms going off, but they're noisy in data. There's so much information flowing at you all all at once. And to be able to interpret it and to make sense of it all is not an easy feat. And what she realized was, like, that was important information. But there's even more important information in just what's hiding in plain sight. And what's hiding in plain sight is the level of concern that a nurse has for a patient. You know, when I was seeing patients and I was in the ccu, when I, you know, I'd make rounds three times a day, maybe four, depending on how acute things were, but the very first thing I would do whenever I'd walk into the unit would be, I'D go to the nurse and I'd say, what's going on? And the nurse would, who's there? On 12 hour shifts, they only treat one, two, maybe at most three patients if it's a, if it's a very non acute icu, but most of the time it's one or two patients. And, and they know, they just, they know the gestalt, they know what it looks like when a patient's about to go sideways. But the thing is, this wasn't about putting the patients into the icu, evaluating those in the icu. This was about documenting patients on the floor and getting seen and put sent to the ICU before things went sideways.

[Dr. Leanne Rosenblitt]

Yeah. And notice what the other property that this kind of observational research has, which is very thoughtful, they picked a measure that did not require any new recordkeeping. Right. This is the kind of dream scenario we have digital exhaust, Right. In this case, it's when the nurse is writing down the notes and how frequently and that gets picked out and turned into signal. And the reason I just want to focus us on this as an example of observational research methods is that a lot of folks, myself included, and our colleagues work in real world data to real world evidence. Forget that. We are all dealing with the fundamental constraints on observational research methods. Observational research. Take, you know, you take what you got, right? And you get what you get and you don't get upset. I've been working on this metaphor that I'm trying to work into a paper and I don't know if it'll work in the paper, but I also don't know if it's worth it in a paper, but it's very colorful. So the way I think about, the way I've been sort of describing the problem of observational research, and it's a specific subset on real world of real world evidence research, is you are picking up whatever the tide left on the beach, right. You're in a situation of hunting for treasure and you're like, man, there's a lot of stuff. Most of it is, you know, driftwood and like seaweed. And sometimes you find something cool. You're like, whoa, there's like an old coin here that the, the tide brought in and you're all excited and there's a lot of cleverness in actually figuring out how to hunt through the seaweed and the driftwood and pick out like the bits of treasure. Right. There's, there's an old musket I found and you know, I'm, I'm just Like, I'm rethinking Robinson Caruso and his island. Right. And then. But there's another kind of cleverness, and I think that Sarah's team demonstrated both kinds, which is sometimes you don't find any treasure, but then you figure out how to pick out the driftwood and like, built that, built the hut. Well, you take the seaweed and you turn it into fertilizer so you can have like, better soil. And you know, it's, it's reusing the materials that you have, but in a different way. And those are two, like very, those are the two kinds of cleverness that I see exhibited here. It's just, you know, it's really brilliant. And you have to like, if you think, really, if you think about it from that perspective. Yeah.

[Dr. Steven Lapkoff]

And, you know, it also makes me think about, you know, another thing which is, you know, as we're talking about real world data and real world evidence, you know, one of the caveats is that unless you understand the data from a clinical perspective, you might treat the data as gospel truth. And to your point, real world evidence is a lot like the pieces that get left on the beach. But if you don't understand that nuance, you may treat it with a lot more credibility than it may deserve.

[Dr. Leanne Rosenblitt]

Yeah.

[Dr. Steven Lapkoff]

And that can be very problematic.

[Dr. Leanne Rosenblitt]

I, you know, I'm pro. May feel like I'm maybe more sensitive to this problem because I'm start. I started from a research, from research methods that was honed in social science. And there we're really used to the problem of very noisy data. And the problem that, you know, the easiest person to fool is yourself. Right. And you're very easy to fool. As Feynman said about, about scientific method, then what we see in, as our ability to process data increases, there's this opportunity for people to do a lot of data processing without paying attention to the methods. Right. So, and when you have data processing sophistication and some statistical tools, you have the, you can have the appearance of creating meaning and order with, you know, what I would call method slop. Right. You just, you know, you didn't pay attention to under what conditions was this data created. What is it seaweed, you know, or is it treasure? And is it kind of seaweed? Is it, you know, should I build a heart out of this? Is it going to collapse if you don't do that? You just because you're getting sort of P values at the end, and just because you have a sophisticated model, if you don't trace it back down to methodological discipline you still have, you still have junk at the end. Right. And I think this sort of brings us back to what AI can and cannot do.

[Dr. Steven Lapkoff]

Yeah. And one of the things that I think is worth pointing out with regard to Sarah's work is that this insight, while it was AI assisted in terms of the work she was doing in terms of interpretation, the concept here wasn't an AI generated thought. I mean, this program was generated through human thought. And for those out there who are concerned that AI is going to replace us all, I think this is a great example where a human idea, a human observation is something that became, you know, the survivability and such that has been proven through the concern project has been staggering. Something like a 37% live. I hope I got that statistic properly. It's at least 33, but I think it was 37. She mentioned in the podcast where patients are doing better, the outcomes are actually materially better because of the observation that this, that was picked up through her work. So kudos, Sarah. I think you did a phenomenal job. And we're not ignoring the AI lift here because you used AI in the course of your research, but I love the fact that this was a human generated. Human generated idea.

[Dr. Leanne Rosenblitt]

Yeah. So, so I mean, the humans enter this, this information, you know, knowledge creation pipeline in two places. The human is the sensor. Right. So the, the, the nurse is like the finely tuned sensor that's sort of picking up on vibes and going, oh, this is, this looks funny. Right. So we're, we're taking advantage of that particular human ability and their understanding of local context. And then the human is also the creator, the AI is the enabler. So the fact that they could develop an AI model to actually process all of this data efficiently and turn it into signal and say, hey, hey, that patient, that, that documentation means that patient needs to go right now, go take them to the icu. Is really, really cool and is only possible because of AI. But I mean, I really like your point that it was human creativity that enabled it. And I actually am, you know, I believe that proper use of information technology tools can enhance human creativity. And here I think this is a good example of it. And, you know, I don't know how Sarah and her team actually came up with the idea, but it was. Damn, it was a good one. So let's talk about Jeff. I think we, you know, we were pretty excited to talk to him because he was our first person who we talked to who was officially in the government. We've talked before to folks who are peripheral, who are doing some, some government consulting. But like, he is, he is the onc. And I think it was so cool that we, he was able to come and tell us how the world is going. And I mean, I think that there's some good properties of hearing from people who work inside the government. You kind of remember that they're trying to do a good job, even though there, it's inevitable that whatever they do is going to annoy a lot of people. Right. You know, us included. But it's not because they're not trying. So it's not always a sign of a competence. It's just that they're trying to negotiate a landscape of rules that are designed to bind behavior and that therefore it's irritating and costly. So what did you think of Jeff? What was the impression that, you know,

[Dr. Steven Lapkoff]

he brought up a number of things which I used to follow the ONC a lot more closely when I was in other jobs and other situations in my career. And, you know, I was much more deep in ONC work when meaningful use was being crafted. I was on a committee back in the day when, you know, trying to figure out how to, how to frame meaningful use. What I found interesting was some of the ways that he was talking about, you know, consent and consent inheritance and chain of delegation, and also some of the things that we discussed in the course of the podcast around how agents are going to actually have similar types of rights by. Through law, through legislation that will avoid data blocking. So data blocking won't be. So if you don't let an agent have access to the data with appropriate consents, you could consider that data blocking, and that could be a fineable offense under the current rules structure from onc. It didn't quite see that one coming initially. And it was an interesting twist on this whole issue of data blocking. I mean, what are your thoughts?

[Dr. Leanne Rosenblitt]

I was, I'm with you in not having paid enough attention to this. So it was actually, you know, I had one of those, you know, Pikachu face surprise moments. I'm like, wait, what they did that I didn't know. So I, I mean, it's a, it's a fascinating move, right? So let's, let's break it up. I mean, we know for that when government acts, right, the stakes have big consequences. And like, things like meaningful use really pushed physicians out of medicine and, you know, the big requirements can you have dramatic things. And so we have to watch pretty carefully when government acts because we got to remember the big consequences can also Happen when government does not act. Right. So it's, you know, to come back to my earlier point, it's, it's their, you know, the function is sort of to try to decide between difficult courses of action. In this case, the ONC said, hey, how are we going to treat these agents? Oh, I know. We're going to give them the same data access rights as patients if they're authorized. Right. So you have, you have this concept of the patient can authorize it. Okay. Is that going to have any unforeseen consequences? I don't know, but I'm going to vet your dollars to donuts it is. I mean, I'm really intrigued by it. I'm not saying it's a bad decision, by the way. This is, I think it's really interesting.

[Dr. Steven Lapkoff]

We don't know.

[Narrator]

It's too.

[Dr. Leanne Rosenblitt]

Right, right. We don't know. But, you know, I'll bet you there's unforeseen consequences. Right. So, you know, I, you know, one thing that comes to mind is, Steve, is what you mentioned, right. Is what happens if a patient authorizes an agent. And that's cool, like I just authorized my agent to go get my data. Then I pass that right on to a third party. A company. That company get gets bought by a larger company. That company gets bought by a larger company which goes out of business and the rights gets inherited by the creditor.

[Dr. Steven Lapkoff]

Oh yeah.

[Dr. Leanne Rosenblitt]

The right this written by particular in a particular way. My right continues down that chain. Eventually some party I never interacted with can get that right. Is the rightfully transferable. Yeah, I don't know the answer. Right. Let's be clear.

[Dr. Steven Lapkoff]

That example you just gave wasn't something Jeff addressed, but I think you're spot on that that could be a potential outcome here. It reminds me of the old line from Spider Man. With great power comes great responsibility. And in the course of how ONC behaves, their moves move mountains that can move people in or out of fields. It can move people in or out of business. And the story is yet to be finished on this one. This has just been written more or less, and we're going to have to watch this space for what's going to come out of it.

[Dr. Leanne Rosenblitt]

Yeah. In a way, I keep thinking about why am I less shocked by some of these developments than some of my colleagues. I'm thinking about the implications of AI I think it's because I just read a lot of science fiction over the last last 50 years. Like a lot. Like what. What this made me think of is the Concept, I think it was Charlie Strauss, it was one of the, you know, great cyberpunk sci fi writers had of intelligent contracts that negotiate with each other. Right. And come up with agreements. Right. And they basically can, can judge whether or not like you need to modify this clause so that, I mean, this rule doesn't quite move us into this world. But think about the concept of, of agents having transfer. Receiving transferable rights.

[Dr. Steven Lapkoff]

Yeah.

[Dr. Leanne Rosenblitt]

Is really interesting because. Because can those agents then transfer those rights to other agents? Well, certain. You know, certainly a legal entity like a company can. Right?

[Narrator]

Yeah.

[Dr. Leanne Rosenblitt]

But anyway, I'm getting, I'm nerding out on kind of the, the contract law here because I, I actually think it's really.

[Dr. Steven Lapkoff]

Well, you are a lawyer by training. Right. So I'm going to give you a pass on that one because. Right.

[Dr. Leanne Rosenblitt]

Yeah. Like I'm cognitive scientist and a lawyer, so I get, I get pretty excited about this stuff. I'm like, oh my God, this is, this is think I've been thinking about. So the other thing that, you know, Jeff talked about that I think has may have a lot of implications is this isn't a rule yet, but he is actively thinking about data blocking applying to writing data to healthcare records.

[Dr. Steven Lapkoff]

Yeah, he mentioned that.

[Dr. Leanne Rosenblitt]

Very, very interesting idea. But again, do we know, you know, how is that going to impact the current landscape? You know, probably from a patient's point of view it's a boon, but it changes the landscapes and what the current vendors. I don't know if we thought enough about that or if, you know, you had a conclusion.

[Dr. Steven Lapkoff]

We should put a pin in this one and come back because I think some of the things we'll talk about in the Zach episode at the end of this are going to direct complicate. They're going to complement some of these.

[Dr. Leanne Rosenblitt]

And also, like, if we're going to talk about patients, let's talk about Hugo campus.

[Dr. Steven Lapkoff]

Yes, let's go to Hugo.

[Dr. Leanne Rosenblitt]

You know, wonderful, wonderful guy. You know, we're so lucky to have him. You know, we've had epatient David and Hugo is somebody who's known in the patient advocacy community as sort of a beacon of how empowered patients can become, you know, and use existing tools to get hands on their data, help with their own care. Right. Just a really inspiring story. But, you know, one thing that jumped out of me in this story that I don't, you know, is that Hugo keeps said a few times that using the AI tools allowed him to do really cool stuff like he was able to retrieve his data and Manipulate it. But he doesn't quite understand what he's built. And that's concerning as a world, as a builder. That worries me. Right. Like there's a tension between empowerment and just constructing black boxes that do magical things that you don't fully understand. And I, I don't know how you felt about it, but that made me really uncomfortable. Like I wanted to understand more and really get into his head about what, you know, what his mental model of that was. But what was your reaction?

[Dr. Steven Lapkoff]

Well, it wasn't quite a black mirror moment for me.

[Dr. Leanne Rosenblitt]

Yeah.

[Dr. Steven Lapkoff]

You know that show from Netflix? Yeah, yeah. It wasn't quite that bad for me, but it was concerning. I, you know, on the other hand, he wasn't doing anything that was particularly worrisome. But it did raise a few other questions to me, which are. And it revolves around the question of democratization of very powerful tools. And it speaks to some of the work we're doing at Harvard around the, around AI literacy and around data literacy. And again, I'm not making any judgment calls here, but I mean, folks who are going to be empowered by using their own data, it's fantastic. They also need to make sure they have the context for what it all means, so they don't make tools that will interpret things in ways that they may not have been intended. And it's a bit of a concern for me on that front because I've got members of my own family who have started to dig into using AI and looking at their own medical data who have drawn 180 degree wrong opinions about their own care and about their own situations. One of whom is very, very close to me, actually, I'm not going to say who, but she's awfully close in my world. And she made a diagnosis of herself once recently where it was completely 180 degrees wrong.

[Dr. Leanne Rosenblitt]

It was AI enabled. Right.

[Dr. Steven Lapkoff]

It was AI enabled. That's exact. Right. And because she didn't have the context to interpret the data from a medical perspective, she certainly was capable from a data perspective. She's a very bright woman who understands data very well. She's very talented. But the clinical aspects of how it all fit together and made the put a picture together for it, she didn't have those concepts down. And as a result, she drew 180° the wrong conclusion. I'm concerned that folks like Hugo who are empowered patients, that I'm hopeful that, you know, the AI keeps them on track and that it helps them avoid those types of pitfalls. But, you know, I think the jury's out on where that's going to go well.

[Dr. Leanne Rosenblitt]

So Hugo has really pretty good systems thinking. So there are things he built that actually demonstrate that he is maybe being a little bit humble about, you know, how the black's boxiness of his building, for example, when he's unsure about something he builds systems where he has two AIs, think about it and come to a consensus. Right. Which is actually a really good model, you know, in the sort of the community, the community of folks who are building modern AI infrastructure. It's sort of a council skill.

[Narrator]

Right.

[Dr. Leanne Rosenblitt]

You build in multiple models with different legacies and you have them fight it out and come to a higher quality decision. You know, he's also was, I think very smart in taking what he built, which was essentially an MCP server that parses data from his health care provider and made it open source. One helps other people, second opens it to inspection and criticism by people who may know more or just have different, a different perspective. That's really, really good. So he's both the beneficiary of the open source community. He basically borrowed stuff that was already open source. And I think he talked to Josh Mandel.

[Dr. Steven Lapkoff]

Right? Yeah.

[Dr. Leanne Rosenblitt]

Who helped them open source guru in, you know, in the fire world and he's, you know, so all of that is good. But I, I mean I have to say for myself, you know, I, I, when I'm trying to build stuff, I feel like I need to have at least a systems level of understanding of how things fit together. Yeah. And so that I can have the ability to test it, know where things are likely to break and I don't need to understand things at a fine grained technical detail. Right. I'm not an engineer. I'm willing to kind of say I can delegate that to a junior developer. Right. So, but, but so I'm bringing my experience in managing IT teams with who have more technical knowledge than I do and I'm applying it to kind of things that I do with AI where you know, you just, you can, you could be stupid and be like I don't understand how that works, explain it to me. But I don't understand is Hugo go doing the black box thing and not asking the AI model to say can you explain this to me? And look, the reason I think he's maybe just being humble is because he did mention to us that he, he had some IT background. I think he was a project manager, he was a software developer briefly, then a PM in it. So he's got stuff to hang that on and I, you know, again, lots of credit to him. He's just phenomenal. But I, I'm kind of wondering if, if he's telling, you know, he's caught up in the story and maybe not quite as, oh, gosh, I'm just slapping stuff together and it works, oh, shucks kind of moment, right? The other. I just wanted to bring one other perspective to this, you know, before we, we go to our next guest. I was talking to a friend of mine, was an absolutely brilliantly insightful psychologist, and he just said something and he's not super technical, really bright guy, right? But not technical. But he's been using AI and I was like, well, so tell me how it's going. He's like, you know, the amazing thing about it is it's the first software tool, power tool, that didn't make me feel stupid. Oh, yeah.

[Dr. Steven Lapkoff]

Oh, I love that one.

[Dr. Leanne Rosenblitt]

Boy, you're not right. You go in and you stare into the command line, or even if you're staring at an Excel spreadsheet, right? And you don't know how to write functions. All of those tools for a novice user are really disempowering because you're typing stuff in and it's always wrong.

[Dr. Steven Lapkoff]

And it's in this place, one comma or one, and it's a mess. Yeah, absolutely right.

[Dr. Leanne Rosenblitt]

And the reaction, you know, the reaction to a person who's a little sensitive is like, oh, I screwed up. Oh my God, again, Again. You know, it doesn't take that. Too many beatings for a person to start. And the LLM models, because they are using natural language as the interface, they're very tolerant of human error. And that's just part of being human. Like, we say things wrong, we type things wrong, we misspell commas, right? Some of us are worse at it than others, but I think it's that tolerance to error, that kind of friendliness to human frailty, is actually what allows folks like Hugo to become empowered and probably to become more technical than they think.

[Dr. Steven Lapkoff]

Well, and it's not just Hugo, I'll tell you that in my own personal world. I'll recap a small project I'm working on. I've got a consultancy as, you know, as my day job, and I recently built a sentiment analysis pipeline. Now, I haven't programmed. I used to be a programmer back in the day when I was a fellow, and I wrote my own code and things like that, but I haven't been really coding in any kind of meaningful way in a long, long time. And when I took 6001, the MIT course on computer science and AI, back in 1993. I found it so daunting that even using the tool called Emacs, which is an editor in Unix, was so daunting that I had to break everything down and go into Microsoft Word and do all my programming in Word, cut and paste it into the command line. And, yeah, you're looking at me like I'm nuts, but that's how bad I couldn't.

[Dr. Leanne Rosenblitt]

Yeah, Emacs and VIM have a steep, steep learning curve. A lot of people break their teeth on that. I know what you're talking about.

[Dr. Steven Lapkoff]

But I was able to write a synthesis pipeline of some interviews we've done with one of my clients. We've got 27 interviews. I've wrote a pipeline to do sentiment analysis, which is actually a fairly complicated analysis. I paid two years ago, ZS Consulting, like, $100,000 to build a similar pipeline, and I built the same damn thing in an afternoon, like two and a half, three hours. It was done and it did it well. So I can understand how, you know, you were saying that this AI stuff is like crack for nerds, man. I'm living that dream. I'm not filming my heating, and I'm

[Dr. Leanne Rosenblitt]

like, you're on the pipe. You're just spending 12 hours in concessions. Yeah, we all go through this. Yeah.

[Dr. Steven Lapkoff]

Oh, it's incredible. And I think Hugo might be there, too, a bit. And, you know, I'm not calling him out and calling it a bad thing. It's great, and it's incredibly empowering. But again, you know, every time, you got to put it all down a while and get some rest to make sure you're doing it right, make sure it comes out with proper response. All these things, these checks and balances need to be there, and I'm sure they're going to get there. Hey, why don't we shift gears into the next speaker? Who was Fred Bennett, now this one. Fred. Just full disclosure. Fred is a. Is a longtime friend. He and I work together at Pfizer, and I am a consultant to his company. His company is called Patient Talker. And he asked me if I'd be willing to have him on a live podcast at AI Tech Week in New York. And I thought, why not? I thought Leon could join me, but his neck got in the way and he couldn't come. Unfortunately, I look like this. Yeah, you did. You were in bed, you were in bad shape, and I felt terrible for you, but we. We went on without you, and we had A wonderful conversation. So Fred is the CEO of this small startup called Patient Talker, which is an AI ambient listening tool. But unlike most of the rest of the AI ambient listening tool world, this one is focused directly on patients. And that's really what caught my attention with this and why I've been working with Fred is that, you know, unlike Abridge or Nabla or Suki and the others in this space, Fred's taken the approach based upon an issue he had with his own family, which was his own, so backing up a little bit. Fred's dad a few years ago had some medical, very significant medical issues and he had a. He would take his mom and his dad to the hospital and to the doctor to be seen. And it's like everybody in the room heard something different. And as we went to publicize this, Leon pulled a frame out of a movie which I'd never heard of and he's chatted me about that ever since. What's the name of that movie, Leon?

[Dr. Leanne Rosenblitt]

It's only because you claim to be a film buff. I was like, dude, it's Rashomon, right?

[Dr. Steven Lapkoff]

I didn't know that film. I never saw that.

[Dr. Leanne Rosenblitt]

It's Akira Kurosawa. But yeah, okay, that was just to tease you. But.

[Dr. Steven Lapkoff]

But Fred lived that dream, which was basically he, his mom and his dad were all listening to the same 30 minute encounter with a doc and came away with dramatically different perspectives on what was said.

[Narrator]

And

[Dr. Steven Lapkoff]

Fred thought this is a big enough problem because his dad left the visit thinking, hey, I'm. Doc said I could have cake. And his mom thought, no, no, no, the doc said, you need to take your medicine differently, Luke. I'm going to make sure you're taking it with the right food at the right time. No, no, I could eat cake. And Fred's like, fred, another perspective altogether. And so patient talked. The idea behind it is that it listens to the doctor patient interaction, it digests the transcripts and, and gives a digest to the patient so they can remember much more accurately what it was actually said. A great use case, honestly. Now I'm sure you could do that with just ChatGPT or with any of these other tools. But he's wrapped the thing around it and it's got a patient facing view to it. And I thought it was really interesting in that, you know, he's taking all this and trying to wrap a business around it. Now his business model needs some work. He's working very hard on that and. But he's got some users, he's got people Using it. But. And I thought from a, you know, practical AI in healthcare perspective, he's living the dream. Like, this is a very, very practically facing problem to solve and he's licked it. You weren't unfortunately, there, but I don't know if you have any thoughts about it.

[Dr. Leanne Rosenblitt]

Yeah, no, I listened to the episode. It was a super, super interesting conversation. And we got to talk to Fred beforehand to kind of get his, get a deeper perspective on it. I think he is addressing a really serious and gap in the ambient AI use cases. I think he's spot on in that being a patient as listener is very difficult. You're sick, you're struck. I mean, I just went through this with my neck thing. I have no idea what my doctor said after the visit. I had to reconstruct it afterwards and then react to it, which was not the best. You know, there's so many reasons why you're cognitively overloaded, Right. Because you know, you're listening to or to a foreign language. Right. They're speaking a jargon. You don't have a foreign.

[Dr. Steven Lapkoff]

You're absolutely. It is a foreign language to most people.

[Dr. Leanne Rosenblitt]

You're spotted, you know, and you're trying to understand the dialect and at the same time parse it into what's meaningful to you. It's a very difficult environment. And I think this is exactly the kind of case where like an ambient listening AI can make it much easier for the patient and go over with them afterwards and say, okay, look, this is what we heard. Let's go over what it is you need to do. Yes, the doctor said you can have some cake occasionally. But that wasn't the point of why he was, what he was telling you. He was basically telling you, hey, you should really eat healthy.

[Dr. Steven Lapkoff]

Right.

[Dr. Leanne Rosenblitt]

So there was a Rochemont at the doctor's office moment. You know, I'm going to have to post the image because we didn't make a normal stream because you guys didn't like it. But I'm posting a separate comment just for that.

[Dr. Steven Lapkoff]

I did not like it. I didn't. I didn't know the reference. So it made it.

[Dr. Leanne Rosenblitt]

Oh, no, I was too esoteric. I understand. Of me being, you know, being a nerd, but so, I mean, I think what it reminds me of is a line that I hear a lot in the communications community and the teaching community is that communication is what the listener does. Yeah, right. So communication didn't happen just because you said something.

[Dr. Steven Lapkoff]

That's right. Yeah.

[Dr. Leanne Rosenblitt]

Right. And then I really also really like this idea, you know, about the, the minimal viable product is not enough in health care. You need a minimal trustable product.

[Dr. Steven Lapkoff]

Yeah, yeah.

[Dr. Leanne Rosenblitt]

I think that connects really well to our work on, you know, seat. Right. Safety, efficacy, equity and trust. Trust is just fundamental. If it's not trustworthy in health care, it is not viable. It's that simple. The V without the T doesn't work. It's got to be a minimal trustable product. So I thought that's really clever. I think you, I think you sort of went to the thing that was difficult about, about Fred's idea. It's a great idea. But the question in my mind is what's the business model?

[Dr. Steven Lapkoff]

Yeah, it's true.

[Dr. Leanne Rosenblitt]

Right. Because we know we don't want to. You don't want to deliver tools that patients have to pay for if it's not funded by insurance or if it's funded by, by the provider, you know, it's not going anywhere. There's no business around it. And so the reason is the question for me is Patient Talker company or is it a component? I can actually imagine it as being a component in a larger health maintenance, chronic disease or care management type of pipeline where the goal is to improve patient's health and part of it is preparing for the visit. But that would also include kind of pre visit preparation, talking to the patient, getting a pre visit interview and delivering to the doctor, supporting them between visits by pointing up to the right specialists. So there I see a lot of value in having that. So I don't know what you think and I don't want to kind of tell Fred what to do, but I'm sure he'd listen. Looking at the entrepreneurial space, which I do follow, you know, because of my work as entrepreneur in residence at Yale and my work with some investor communities, that feels more like a, like a more natural fit to him for, for him. You know, I don't know if you

[Dr. Steven Lapkoff]

agree, but not only do I agree, but he's already been given that, that, that advice. I, I gave him that advice several months ago that this is, you know, put it on the roadmap because I think you're right. I think this is a piece of the picture. I don't know, it's the whole picture, but from some other stuff I've done in my career, I can absolutely see this being a component of a bigger, of a bigger system and you know, hopefully he'll get there and we should come back and check with him in a year and see where, where patient

[Dr. Leanne Rosenblitt]

Dockers gone, you know, when, when I teach the Data Driven Value Creation healthcare course at the Yale School of Management. One of the hardest lessons to impart is where does the money come from? Right in, in, in healthcare. And if, in this case, if there's no billing code, there's no money. And if, but if you make it part of the care navigation pipeline, there are billing codes for that.

[Dr. Steven Lapkoff]

That's true.

[Dr. Leanne Rosenblitt]

So there's money. Right now you're like, well, they just got to figure out how to do this in a way that's billable to the insurance company. Yeah, I mean, it's, it's a. Not all of our. Many of our listeners aren't really thinking about the business side, but those, those who are like, remember, that's a very, very big, you know, gap in healthcare entrepreneurship. But, yeah, we should move on.

[Dr. Steven Lapkoff]

So let's move on to our conversation with Zach Kahani. And for those of you on the podcast who don't know, Zach is one of the foremost minds in AI and healthcare and has been for close to maybe 40 plus years. He started his career in Boston and was at MIT and working with a guy who I know, who was also my mentor, Pete Sulvitz. And he eventually wound up, even though he went and did his residency in pediatric endocrinology and became a doc in that field, he drifted into the informatics world and has never really looked back. And he had some of the most interesting things, I think, of the five podcasts we're reviewing today. I loved what Sarah was great, no question. But Zach makes you think we love

[Dr. Leanne Rosenblitt]

all our guests and our children equally.

[Dr. Steven Lapkoff]

Yeah, we do. It's true. But Zach is Zach, and I think you and I both know him from our work at Harvard, and he, you know, why don't we unpack some of the things that he was talking about?

[Dr. Leanne Rosenblitt]

That's all right. Zach is one of those minds that offers such fresh insights, it's phenomenal. It just reminds me of the old joke about, you know, the wise old fish is swimming in the ocean and comes across two younger fish cavorty. He says, hey, boys, how's the water today? Swims away, and one young fish looks at the other and says, what the hell's water? Yeah, right. So talking to Zach, I think, has that quality, right? Because he will ask the most profound question, and you go, what the hell? Why didn't we think of this? Right? So I think that that quality, I think, showed up in his. The work he's probably best known for, which is asking the dumb question about why can't. Ehr, functions be like apps on the iPhone.

[Dr. Steven Lapkoff]

And he asked it early on, that was the question.

[Dr. Leanne Rosenblitt]

And he asked it, he was like, I don't understand. Why can't it be, why can't we do that? And people are like, we don't know, let's do that. Right? And it resulted, it basically resulted in Smart and Fire, which is the packaging standard for having healthcare apps operate a bit more. Not quite, but much more like apps would be on your phone. That rather than disassembling the whole damn EHR and bolting in some giant component, right? Which is what things were looking like before. So that was just an interesting bit of background story for why he was building. But the other one, and the one I really wanted just everybody to follow and maybe super interested is the question he asked about AI models. He said, but we're building all these AI models in healthcare. What are their values? What values does their behavior represent? I mean, we all care about this, right? It's part of the alignment problem, and that's been discussed a lot in AI safety literature. And everybody building a lie is really sensitive to when we build models, we need to align them to human goals. But I think expressing it in terms of values and what's getting built into these systems is such a powerful lens because these systems are behaving as if they had values, right? And that's one advantage of thinking of them as agents is where you get to apply psychological concepts. We think of agents as entities that have beliefs and desires, right? And the desire to come of their desire represented is what, by what they value, what values they follow. The trick is you can't ask a person what their values are. Or rather you can, Sorry, but you won't get accurate answers. You'll get, you know, you'll get confabulations or what if an AI, we call that hallucination. But you can measure what their values are by giving them choices. And that method, right. What Zach said, said is like, well, we can't just ask them, but let's measure them. Let's give them choices and see how they go boom. Mind blowing, right? The water's fine. I didn't know there was water there, but there's water, but I don't know what jumped out at you.

[Dr. Steven Lapkoff]

Oh, I mean, this was such a profound question to ask. I was just blown away, you know, that I, I, I've been doing another podcast with my rabbi, a completely different podcast, nothing to do with AI, but we, we had him come and give a talk at the DCI Network at The very beginning of our AI journey around bioethics. And the question we posed back in that time frame was what about the bioethics of how AIs are going to make decisions? And this was asked and answered back in 2023. Very, very early on. We didn't have any question, any, any answers at all. All we were doing was posing the question. Zach posed that question and is starting to grab some answers. And the question is, when there's a life or death decision, what values will the AI use in making recommendations to clinicians or to patients for that matter, in terms of what they should do next? And the question is that, and this is a subtlety which got my attention as well, which is that AI models are built with data. The data that it's built with in healthcare system tends to come from EHRs, tends to come from real world evidence. But what's the problem with that? The problem with that is that that data is intrinsically what's baked into that information are our biases, are our values, what transpired and when and why. But you don't know it. You don't exactly. You can't tease it apart. It's like baked in. It's like trying to take the sugar out of a cookie.

[Dr. Leanne Rosenblitt]

An interesting thing about beliefs and desires is that in some ways beliefs are more explicit.

[Dr. Steven Lapkoff]

Yeah.

[Dr. Leanne Rosenblitt]

And easier to define. And desires can be hidden.

[Dr. Steven Lapkoff]

Yes.

[Dr. Leanne Rosenblitt]

Right. Our values, values can be cryptic and sometimes you're cryptic by design. Right. I mean, I don't want to get into like Freudian unconsciousness. I just think that people are hypocritical. Right. We sort of, we do things that we say we probably shouldn't. Right. All the time. And we look at, we believe that we ought to do things and we don't do them. And I actually think that making rules explicit is often very uncomfortable because it reveals the hypocrisy of both ourselves as individuals and the organizations we serve. Think about how hard that makes designing systems in AI that represent human values. Because when you're building an autonomous vehicle, you have to declare in advance what choice you would make on a trolley problem. Yeah. Or the version of the trolley.

[Dr. Steven Lapkoff]

Define the trolley problem for the audience. Some people may not know it.

[Dr. Leanne Rosenblitt]

Okay. Yeah. I mean, so the trolley problem is a, is a, is a well known problem in moral philosophy where, you know, there's a, there's a trolley and it's going down and there's. For reasons that we will not explore, there are people tied to the tracks. On, in, on, on one track, there's there's one person, and on the other track there's five people. And in different versions of the store, of the, of the trolley problem, it's going down to kill five people. Will you pull the lever to redirect it so it kills only one person? There are fascinating psychological variants on this where, like, you have to push a fat guy off a bridge to stop the trolley, right? And you can actually demonstrate that Jonathan Haidt is, let's say, a guy who does beautiful, beautiful research in moral psychology who's demonstrated that the qualities are rare, that vary how people react are not like logical inference about utility. A lot of his, like, psychological, you know, forces about, like, hey, hey, you can't push a guy down. But, you know, pulling a lever is okay because it's a little bit removed. So there's some really, they're interesting variants that we don't have time to explore, but the version of that you, that you have to consider when you, when an atomic autonomous vehicle is. It's going down a street and it has a choice of killing a pedestrian or veering off into the sidewalk and killing five pedestrians, or vice versa. And you got to build that in, right? We don't have to think about it. It's a heuristic. We just, you know, humans just do it. We do it. We, we value life implicitly. We've, you know, we decide to make those decisions without stating them as rules, because stating as rules is really uncomfortable. But I don't want to run off with it. I, I just think that's one of the reasons, you know, I suspect that this exercise becomes really, really fraught. But I, I still think it's incredibly what, what Zach is proposing to do is really, really important, which is look at the models that are being put out there and measure how they make those decisions.

[Dr. Steven Lapkoff]

You know, one other thing that Zach talked about in the, in the discussion happened to be around how do you protect the AI models? Because at the moment, there's a new, A new threat out there. We're all familiar with phishing threats, and we're familiar with hacking threats, but now there's threats that can actually change how AI models behave. If you bomb it with enough information of one type or another. And this is a pretty subtle thing, but if you send enough information to an AI model and it's reading the information as a signal, and that signal is being repeated in a disproportionate way, that signal may get picked up as ground truth. And, you know, again, this one, the story is not done on this one. This is the beginning of the story on this. On this challenge. You know, I don't know where that one's going to land, but Zach discussed this in the discussion along the discussion. And, you know, I know you've been thinking about this one, Leon. I think that this is a new world risk, and I don't know that we're necessarily prepared for it or the models are prepared for it.

[Dr. Leanne Rosenblitt]

Yeah, it's. So we've not just been thinking about it, we're experiencing.

[Dr. Steven Lapkoff]

Right.

[Dr. Leanne Rosenblitt]

Well, like, we're on this podcast, so we get marketed to a lot. And one of the things that we get marketed is like, make your podcast visible to all AIs.

[Dr. Steven Lapkoff]

Right.

[Dr. Leanne Rosenblitt]

Like, you got, you know, search engine optimization isn't enough anymore. You have to have AI optimization. Like what? You know.

[Dr. Steven Lapkoff]

Yes.

[Dr. Leanne Rosenblitt]

And of course, commercially speaking, it's true. Right. More and more of us are using AI to search for products. Right. I mean, I don't go to Google anymore. I've got my product search pipeline set up. It already knows me. It knows the kinds of things I hate, you know, and it's. It's designed to kind of give me. Give me what I want. So obviously, this has to evolve. There's a technical question that I think about where this bombing can happen. You know, one is it could happen on, you know, early in the process, during training. Right. And then, you know, and that feels a little bit more dangerous. There's screening mechanisms that, that can prevent it from happening. You could have good engineering approaches that prevent it from happening, or it could happen during. During inference. Right after the model is built. You, you know, it's going out to the Internet to search for stuff. I really think that Zach's approach is a solution to both. We don't know. You know, first we have to have good processes that protect the training, and we have to have intelligent screeners. But ultimately, we have to measure the model's behavior. And, and, and I think that's. That's the answer. Right. It's. It's the monitoring of the models. And, you know, on that, I think we probably need to bring this to a close. Yeah.

[Dr. Steven Lapkoff]

I will just say one last thing before we do close. This is right up the alley of the work we've been doing at Harvard, keeping an eye on the model and model protection and making sure that the models are intrinsically monitored and safe. Those are things we've been discussing, and I'm working on paper right now with Dean Sidig around model protection.

[Dr. Leanne Rosenblitt]

In any event, who shall watch the watchers. It's us, right? We apparently volunteered.

[Dr. Steven Lapkoff]

There you go. Hey, Leon. This has been a fantastic discussion. I hope that our audience enjoyed it as much as I did. Really thoughtful stuff from all of our guests. Let's hand it to you for fun.

[Dr. Leanne Rosenblitt]

We love equally. Remember, we love equally. You want to clear this out? Yeah. No, this was fun. Steve. Let's do this again soon in another five episodes or so.

[Dr. Steven Lapkoff]

Yep. Sounds like a plan. And we'll see you all again next time on Practical AI In Healthcare. Thanks for listening.

[Narrator]

Thank you for joining us this week on Practical AI in Healthcare. If you're ready to go beyond buzzwords and hype and explore how AI is truly transforming healthcare, stay tuned for more conversations that get us to what works. Until next time, stay pract.