← Advancing Health
Loading summary
Transcript23 lines
- [00:00]
A
Welcome to Advancing Health. March's Leadership Dialogue podcast explores how a collaboration between Rutgers Health and RWJ Barnabas Health is unleashing the power of AI carefully and methodically to improve patient safety and reduce clinician burnout.
- [00:21]
B
I'm Mark Boom. I'm the president and CEO of Houston Methodist and the chair of the board of the American Hospital Association. So I want to continue the thread of our discussions. This month we're going to focus on innovation in patient safety. All hospitals and health systems we know put safe, high quality care first for their patients. And for decades now we've been using innovation to improve outcomes. And we know that we've seen really dramatic improvements. But we also know we can never be complacent. We need to continuously work to advance safety and quality because we have a sacred responsibility to keep our patients safe at every single step. Whether it's our physicians or nurses who are at the bedside, or leadership shaping system wide decisions, we always have the same goal, which is be safe, deliver safe care. And innovation is a critically important tool in making that happen. And thankfully we have a lot of new tools that help that happen. So for example, small wearable devices that can monitor vital signs in real time and send updates directly to nurses and giving nurses more time at the patient's bedside, patients more time to recover and less sleep interruption. Adopting innovative approaches is really, as I said, critically important, but it sometimes feels pretty challenging. And so I'm very excited to have a guest with me today who is expert and really doing exactly the kinds of things I was just talking about. So please join me in welcoming Amy Rockman. Amy is the director of the Artificial Intelligence center of Excellence, which is a system wide collaborative initiative between RWJ Barnabas Health and Rutgers Health. Amy, welcome. So Amy, I want to begin by asking you to share a bit more about the partnership. I understand that your mission is to dedicate responsible development and integration of artificial intelligence to improve patient care and also a goal of reducing clinician burnout. So tell us a little bit about how it came to be and why it's notable for the work you're doing.
- [02:14]
C
Great, thank you so much, Dr. Boom. So we started this center, this group, a few years back. Some forward thinking leadership was really seeing the potential of these powerful AI tools. And so what we created is essentially an AI focused learning health system. So that's a system between our university and our health system in which research is informing practice and practice is then again informing research. And so the idea between these two structures and bridging them Together through the center, is to bring those research experts together with our everyday heroes, real clinicians in the health system, practicing medicine, so that we can better inform the tools that we're introducing and how they can really drive change throughout our hospitals. So we brought together these two different sides of the health system and the university, and we did it with AI because it really requires this next level focus. When you're bringing in and integrating these powerful artificial intelligence tools, there's so many things to think about from a safety perspective. There's safety and security, of course, then there's validity and reliability of the tools, and that's with a lot of the technologies that you're bringing in. But AI introduces this whole new layer, since there's so much about it that we still don't understand. So explainability, for example, and transparency, interpretability of the tools, all of this we're still learning. As AI is coming out, AI is looking at these huge, sweeping statistical associations, and it's so incredibly powerful. It's able to do AI incredible speed, accuracy, so many changes that come with the tools, but we need to be able to understand them, validate them, evaluate them. So there's actually a whole AI product life cycle that we started to follow, and the Coalition for Health AI has really created this in detail, and it fit very seamlessly with our work and how we think about. How do we determine which area of our health system would most benefit from a tool right now? How do we then identify a tool? Is it going to be homegrown internally, the university, or is it going to be vendor acquired and introduced? Then once we introduce it, there's a whole integration process of integrating it both technically into your infrastructure and into your clinical workflow. Then you need to monitor it, fully evaluate it, identify gaps, and the process restarts. So as we're following this AI lifecycle at each step, there's a lot to think about. And so it's not just so much that you need to think through, it's how interdisciplinary the work truly is. So how many people you really need on the team to be able to think through this in the most impactful way and in the safest way for our patients?
- [04:57]
B
I hear that it sounds like you're extremely intentional in how you're approaching this. I mean, you're not just sort of waiting for things to come to you. You're sitting there saying, what are the problems we want to solve? And now how might we build something ourselves or go look for a solution? Is that corre.
- [05:12]
C
It's actually incredible how there's multiple different wavelengths coming together to make a lot of these decisions. And so a lot of it starts from our KPIs and drivers and risks and thinking through. First we started introducing, for example, administrative tools as they were much lower risk and there's still a lot of high reward even for patient safety. If you're able to catch a lot of those documentation issues, you're able to address those, you have better documentation for your patient, you have a better patient history. So we introduced some of these low risk tools first and then started introducing the more high risk tools. We also, we started introducing it by again, looking at, you know, those KPIs, those drivers, our verticals, our horizontals. But as we're doing that, we're building these interdisciplinary teams. And as we're doing that, we're starting to learn from the teams and really get a deeper understanding of how the AI tools we've started to introduce are affecting the clinical environment. And so now we're getting a grassroots input as well. And so the decision making is really, really thoughtful. It involves a great number of people and a great interdisciplinary effort.
- [06:17]
B
So I know a lot of people would like to follow your lead and do things alongside. Can you walk me through an example of something you've tackled and how big is the core personnel versus the interdisciplinary team versus getting to the grassroots? Would you walk me through kind of an example of something that that's worked and how that has been put together?
- [06:34]
C
We've introduced dozens of tools at this point and some of them really have taken these incredible team efforts. So I'd love to give you an example of one. And so I think the AI enabled Clinical Deterioration Index is an off the shelf EPIC tool that we introduced into RWJ Barnabas Health. And we introduced it starting with a small pilot and it required a large interdisciplinary team of providers, administrators and tech experts who are really working, coming together on a weekly basis at one point to really review as you introduce this tool. And so let me share what the tool is. It is a early warning system for clinical deterioration, flagging a patient for potential deterioration 24 hours before the deterioration is expected. And so we all know that earlier intervention in many of these cases is essential. And so it's really a game changer to be able to have that much warning and be able to make a change and actually impact the care. And you can impact the care in different ways. In our health system, we chose to impact the care by moving that person to the ICU in advance. Other health systems have made different choices, but you have a choice that you can make and that's what matters. You can really respond sooner. And so in order to, to do this though, and to make it work, a lot of thought needed to go into it. Because even though these products, many of these products seem, they're off the shelf, they should be easily implemented. They might be easily implemented into your technical infrastructure, if you have epic, for example, but that does not necessarily mean they're easily implemented into your work streams and your workflow. And so when we first implemented it, there was so much to think about as far as who is getting the flag. It's a rapid response team. How are we adjusting this team? How is that flag getting to the providers? And then we're looking at, constantly at the sensitivity and specificity because we're getting false warnings. You know, we want to ensure we're not missing warnings. And so how do you adjust the algorithm when the algorithm is a complete black box? Most of the algorithms that we get, even when they're data analytics focused, we don't know everything about it because it's proprietary. But in AI, it's truly a black box. In many of these situations, we don't know at all how the AI is getting to the answers that it is. And so we need to create our own interpretability layer, or explainability layer, if you will, to really try to understand. And so when we did that, we started to stratify and we started to see that there are different proportions in our population than in the population in which this was initially trained. And so we can make some adjustments. We made some adjustments for hospice, for example, and we removed some of the stratum and we found that we could adjust it and really get it to an ideal sensitivity and specificity where now the 24 hour flags were so meaningful that we saw an over 18% reduction in, in hospital mortality.
- [09:21]
B
Wow, that's very impressive. So that really meets that noble goal of, of, of what you're talking about with this. So when I've read up on, on your center, and I think you already gave us an example, but give us a little more around kind of this idea of a living laboratory. What, what do you mean by that exactly?
- [09:36]
C
Yeah, so we're, you know, exploring the world and we're doing this work right in that real world health care setting. And so if you think about how we're moving research from bench to bedside, most of the work really is focused on that bedside space of integrating directly into the healthcare system. But as I mentioned, the AI lifecycle earlier Right. It comes back to the bench, it comes back to homegrown at certain points. But it's a living lab because we're doing a lot of this evaluating and studying and all of this work together, interdisciplinary work, work in that real world space. And so what ended up happening is that we brought these interdisciplinary teams together to integrate into the workflow. We also brought the interdisciplinary teams together to evaluate afterward as part of that life cycle. And as we started bringing these different expertise and areas together, naturally a research hub formed. And so you started to have everyone that I just mentioned who's in the health system trying to integrate and look toward those dashboards and those analytics and really make adjustments in the clinical workflow. And now we're also introducing engineers and computer scientists and statisticians who are going to look even a little bit deeper from a research perspective now that we've fine tuned to a certain degree. Let's look even deeper and really study and validate and ensure that we really know what we saw isn't due to confounders. What we saw is real. Right. That 18% drop is a real value that we're seeing and that we took offline into a lab, studied it even further. Once we have findings from that which currently, for the Clinical Deterioration Index, we have a publication under review with NEJM AI where we looked into all of
- [11:15]
B
those indicators you're impacting patient care and patient safety and at the same time studying it and having the discipline to really make sure that it is indeed your interventions that are doing that and then sharing it with the rest of the world so we can all move the needle forward. I mean, it's really wonderful the way you all do that. So, you know, give me a couple other examples of some things you're working on these days.
- [11:38]
C
Yeah, there's so many different tools and technologies out there and there are so many different areas where we're really trying to expand and understand this technology further. So we also introduced some different platforms that are ambient AI, which is really popular right now because it makes such a difference in our ability to practice medicine with our patients.
- [11:57]
B
Yeah, you can count me as a fan. I use it in my primary care clinic. I love it.
- [12:02]
C
That's great. Exactly. If you can have a tool that can record your conversation so that you can interact with the patient directly, then it's a game changer. And now they're even, you know, they're advancing so rapidly able to take those notes and actually input it into the system for you. Now your documentation is potentially even better. Than before. But with all of these tools as we're introducing them, you really do need to think through those strengths and limitations. That's where that living lab model really comes into play. Because as we're introducing this, you can't take human in the loop out of that one. Right. So at the moment you have your, you know, your DAX analytics, your bridge, all of your different vendor products that can do this ambient technology. When you get your notes back in your practice, you need to review it, right? It's like you got a trainee, right, who's working on it, and they're great and they're amazing. But if you don't review those notes fully, something will get missed potentially, and that could impact the patient safety ultimately. So making sure human in the loop is there, especially as we move toward more advanced AI types. And so there are a couple different ways that we're doing that. One is that as we start to build these homegrown technologies, we're moving toward agentic AI. And so now the AI is not only generating content, the AI is taking autonomous action potentially. And so human in the loop has become more important than ever and ensuring that where that's needed, the human in the loop is still there and that there isn't a problem of overreliance. Right. And that we're trying to reduce bias to the algorithm, but by reviewing thoroughly from a traditional practice perspective as well. Then there's also, again, as mentioned earlier, the explainability and transparency of the products themselves. And so we are trying to understand better because some of these tools are so powerful that we're introducing them due to the changes that we're seeing. So we see, you know, that 18% drop in mortality and it's worth introducing that tool. Right. But we also want to know how the AI is getting to the answers that it is. And so we're starting to think through in our AI learning lab, how do we actually make these tools more explainable and starting to work with the vendors on how explainable is this tool and can we get there? Do we only have post hoc methods where we're looking at heat maps, do we have anti hoc methods where the AI can actually show me its work the same way that you would ask a person, a trainee, a resident, to show theirs.
- [14:10]
B
Often hear that part of what AI is doing these days, nobody really totally understands in terms of some of that black box. So that, that I imagine can be a little bit of a challenge. What you described there.
- [14:21]
C
That's right.
- [14:21]
B
If you could tackle something, what's the big Something you'd like to tackle coming up.
- [14:26]
C
There are so many different opportunities here, and this area is moving so fast. Everything is moving so quickly at lightning speed. And there's so little that we know at the moment. Right. We don't know. For example, there's not a lot of information about how this impacts your roi when you first go to choose a tool, there's not a ton of information about how it might affect your patient population as you go to pick this tool. All of these, you know, you need, to some degree, take a leap of faith and you need to invest in these tools. But these tools are the way of the future. And as we've seen, they're so incredibly powerful. And so I think one thing that we're working on is how do we maximize the strengths of these powerful tools while minimizing the limitations. Right. And in many ways, it's both dual about how it's designed and how it's used. Right. So we're introducing, for example, AI chatbots, or if you're using automated response technologies, speaking to a chatbot could seem like it's more empathetic, for example. Right. Because it never tires. Or speaking to a chatbot could seem less empathetic because it feels like a robot. Right, right. So how that tool is designed and how is that tool is used makes such a difference. Same with the LLMs. Right. So Open Evidence was released not so long ago, and it's a super powerful large language model to be using in the clinical setting. But it really depends what prompts are entered into that.
- [15:41]
B
Yeah, input matters.
- [15:42]
C
Yeah, exactly. Prompt engineering is an entire study of itself now and what kind you're going to use. Is it going to be one shot or zero shot? You know, is it going to be structured? So training the next generation of providers to understand how to use these tools properly is a huge area for us. And how do we think through that? How do we essentially ensure to the best of our ability that the tools are being used in a way that does minimize bias, that does minimize. Minimize over Reliance. MIT just came out with your brain on ChatGPT study showing what a big cognitive debt you're seeing if there is over reliance on the tool. And so we're trying to avoid that by now educating the next generation on how to use this and by educating physicians that are in the hospital at this moment and are starting to get these tools. And I will say that we've managed through this center, through this structure to drum up a lot of excitement about these tools. So we're Seeing a lot of the providers are coming to us, eager to get more and more and more of the tools. And so that's great, that's a great place to be. And people are very interested in working on these interdisciplinary teams together, which is really important. But so the key now is to ensure that every time we adopt one of these tools, we've thought through the process, we've thought through that AI lifecycle, we've thought through how the providers are going to interact with it, how are you going to use it, we've thought through how is it designed, do we have a set sense of what the bias is for this tool, do we have a sense of what the explainability level is for this tool? And so we know to the best of our ability what we're acquiring and integrating into our health system. And we have an expectation of this powerful tool. What will be the change, the transformation? We'll see. And then super fun part for me, I'm an immunology background, is we're monitoring it and we're ensuring that that really happens.
- [17:23]
B
Again, I'll say I love how structured and thoughtful you are and how you're linking it to all that and now education as well. I mean, you, I know you have many residents. This is bringing up the next generation of physician residents as well as obviously nurse trainees and others, which is great. Well, let me ask you one closing word, if you had some closing thoughts to the colleagues who are watching this. You know, you all have a very impressive center. Not everybody's going to be quite as far along, but we're all on this very fast moving train. What would you say to those individuals about how to embrace change, how to invest in innovative tech technologies? What would be some key messages?
- [17:56]
C
Absolutely. So communication is key and being honest, showing the excitement, the potential of these transformative applications, but being practical about it, it's not always going to be easy. You're not going to see that transformation potential right away. I think some of the ambient technology is a great example of that. It also required a lot of tweaking before. People felt like the output was to the the same level as their own notes and that they would take it without the drafting. Taking more time than if you just had written them on your own. Right. So being really practical about that, but being supportive and excited. This is the first generation of these tools. Right. We really put the investment into this. You'll see as they continue to grow, just more and more powerful to support our workforce. And that's a key piece of communication too. In a messaging is that they AI is here to support and enhance our workforce, not to replace it. And it has been enhancing it. You can tell as you talk to a lot of the providers who are using it, they're excited. It's meaningful. There's change happening that makes them feel like they can have the joy at work back again. That makes them feel like they can really take care of their patients in a way that felt like it was gone for a while. And these tools are there to make that difference in medicine.
- [19:09]
B
I love that at Houston Methods we have kind of two overarching principles around new innovation and the work we do. And one is obsessively focus on the needs of our patients, the communities we serve. And then a close second is improve the lives of the people caring for those patients and connect them to the things human beings can do. You know, take away some of the drudgery and other things that prevent that. It sounds like we're on very similar page. So anyway, thank you Amy for your time today. What you're doing is really, really very impressive, very inspiring. And I know you all are already making it difference in people's lives and I can't even imagine as this promulgates across the field and profession. You know, we all share that goal of keeping patients safe, keeping people at the center of everything we do. So thank you again. Thank you everybody for finding time to listen and I will be back in another month for another leadership dialogue conversation. Thanks so much.
- [20:01]
A
Thanks for listening to Advancing Health. Please subscribe and rate us five stars on Apple Podcasts, Spotify or wherever you get your podcasts.