← Practical AI in Healthcare
Loading summary
Transcript81 lines
- [00:02]
A
Welcome to Practical AI in Healthcare, the podcast that cuts through the noise to spotlight real world solutions delivering real world value. From patient care to clinical research, from life sciences to patient engagement, we focus on what truly matters in healthcare today. No hype, no theory, just practical insights where AI is making a true impact. Dr. Steven Lapkoff and Dr. Leanne Rosenblut are your hosts as we explore what's real and moving the needle in this exciting new domain. Welcome aboard and let's get to it. This week we're going to dig into an area that we actually haven't touched on before. In our podcast, we're going to talk about wellness. We're going to talk about how AI and wellness kind of get together. And we have a guest on the show today who's knee deep in that space. Her name is Renee Dean. She has spent over 20 years building AI that does things that we don't really think about at times. And generally we're thinking about how to help doctors do better diagnosis, how do we repeat studies, how do we get data that's replicated. And Renee's working from the other side. She's working on how to give patients some empowerment in terms of their own wellness and use their own data to help with that. So, Renee, first of all, welcome to the podcast. Before we dig in, for those who haven't heard about your organization, Inside Tracker, maybe give us a 30 second version of that and then I'll get into our main question, which is how did you get to where you are today? So start off with who your company is and what they do and then we'll get into your origin story.
- [01:34]
B
Sure. Great. Thank you so much for having me here. It's wonderful. So I work at a company called Inside Tracker and what InsideTracker does is it provides highly evidence backed and that's very important. And I'm sure we'll dig into it later. Lifestyle recommendations, and by that I mean non prescription interventions to users in a personalized way. So we will take information about a user's health state and that can be via asking you questions about your diet and exercise regime. It can be via fitness trackers like a Whooper or our Apple Health Kit, also through clinical biomarkers, all of our standards that we're familiar with, like your lipid panels and your metabolic panels, as well as DNA information. We combine all of that together with a knowledge base of thousands of clinical studies that look at different, you know, ways you can eat, exercise, meditate, for example, sleep or supplements that you could take and match those up Such that we come up with a personalized action plan that's based on clinical studies that are relevant to your biology at that time.
- [02:51]
A
That's really interesting. And yes, we're going to dig into this in some detail in a second. But how in the world did you get here? You started off not in consumer health, you started off as a molecular biologist, right? And cell biologist.
- [03:03]
B
Yeah, yes, I did. So I did. My training is really in engineering. I did my undergraduate in bioengineering and then I did a doctorate in molecular and cell biology. And that was very hands on basic research. I spent I think six years squeezing eggs out of frogs and looking at spindles and how chromosomes condense. So that's what I did my dissertation on. But interestingly, and it's, I would say that when I was graduating from graduate school, that was when microarrays were really hitting the scene. And so we were so excited about the idea that you could just do this one experiment and get like 2,000 data points from it. And then what everybody would do is they would pick their two favorite proteins or MRNA's that they knew about and then they would further characterize those and leave the other 1998 sort of in the dust. And so after graduate school I was looking for jobs and I saw this wonderful position at this company called Genstruct. And what they were doing was they were trying to holistically analyze those 2000 significant state changes you might get from a microarray and put that holistically into a pattern. So I spent 10 years at that company and that's really where I got my foundations in computer science and AI and really using knowledge to help interpret large amounts of data. So I started there. Yeah, that was, that was, that was the first one. And then I went to a handful of different, other different companies, but it was always in the realm of personalized medicine. And Inside Tracker is really the, my first foray into non prescription, you know, interventions, which has been a very different and interesting experience for sure.
- [04:54]
A
So you're, you basically got into this in the early 2000s. You've built an AI reasoning engine, you've rebuilt that was used now for consumer health. Help us get from there from where you started to, into this and how you made that transition. Because a cellular biologist from a wet lab into this space, it's not a straight line. Or is it a straight line? Why don't you help us understand.
- [05:19]
B
I would say that the biggest leap happened between graduate school and that, that first job because I really went from being a Cell and molecular biologist. And you know, I spent six years thinking about one particular protein complex to jumping into systems biology and then becoming a generalist about biological pathways. And at that point, really the type of work we were doing, and I will mention a gentleman by the name of Dexter Pratt, who was really a lot of the brainchild behind the platform that was built at this company, Gen Struct back then. And what it was essentially a knowledge representation and reasoning type of artificial intelligence, which is purely white box. And what it allowed us to do was to take everything that we knew about transcriptional regulators of MRNA's and curate that information into a knowledge base. So for example, if we had, let's say a drug that knocked down pi3kinase and then you did a microarray experiment and you saw that there were 200 genes whose expression changed, you would take that information, you would curate it in this computable format in the knowledge base and then you would know that pi3 kinase was upstream of these different downstream regulators. So just imagine that at scale and you curate every single microarray experiment that had ever been done into a knowledge base that allows you to take a new data set with different MRNA changes and say, what does this look like? So it allowed us to really build these kind of beautiful biological pathways that allowed us to characterize the biological mechanisms of disease. If we had response versus non response data from a drug trial or something like that, then we would use that to characterize the deep biological mechanisms of response and non response. And it was fast, right? You could just upload the data in there and out would come these hypotheses as to what was going on. And then we would have scientists really go in there and dig in and try and understand what was going on and truly build either disease models or models of response and non response. So I had met this, I worked with this guy, Gil Blander at Salventa or Jens Direct at rebranded to Salventa. And then Gil went on to the founder of InsideTracker. I had known about an inside tracker since its inception, but I actually wasn't part of the company until about four years ago. But when Gil brought a lot of this thinking that we were doing at Gen Struct over to Insight Tracker and wanted to apply some of those same principles of this white box causal knowledge and using prior knowledge to help interpret data into more of a like a non prescription world.
- [08:23]
C
So Renee, it's funny, you were giving me flashback to the early microarray data storage problems actually. So you and I sort of Sort of share the early career trauma of working with microarray data. And I remember I just started building large scale data systems when that Illumina push out this like first 400 marker chip and like within 5 years we went from like 400 to like 5000 to like 500,000 to like 2 million chip. And none of us knew what to do with the data. We're like, what are we supposed to do with this? Like the storage was like the storage, intelligent storage was a real problem. So it was funny, you reminded me of that deep history, but I also remember those heady days of trying to organize genomic data connected to biological outcomes to inform systems biology type questions. So I think that's a really interesting, you know, deep background to tackle the kind of problem that you're handling now with, well, with wellness, right, which is traditionally so messy. So let's help our audience define the problem a little bit more precisely.
- [09:27]
B
Right?
- [09:27]
C
Let's start with what the typical person gets. So a normal person gets a lab panel and a physicist tracker readout, right? Like, like I'm looking at my iWatch and I'm checking my, my iPhone. What actually goes wrong today? Why is turning the data to trustworthy action so difficult?
- [09:45]
B
I mean, I would, I would say that the first challenge isn't even anything that goes wrong. It's just that we're inundated with so much data, right. And it's hard to interpret it. And let's just take fitness trackers as an example, I think, because it's not, they're decidedly not in the clinical space compared to, let's say, clinical biomarkers, where at least you have a doctor that can help interpret things. I mean, it's great that we can track patterns in our resting heart rate, but what is, you know, what does that actually mean? So my resting heart rate went up by three beats per minute last night, higher than normal. Like, is this a real problem? What was going on? You know, help me, right? There's that and there's, there's all these recovery scores and heart rate variability and you know, my sleep duration, my REM and deep percentages. Like, what do I, it's great that you can tell me what my REM percentage was last night, but it's not really helpful unless you tell me, is that good or bad? And if it's bad, what do I do about it? So, so I would say that, you know, a lot of what we do at InsideTracker is exactly that. It's, it's, let's take this information that we have about you, put it in context. So, for example, we do have optimal ranges. You might go to the doctor and you'll get your LDL cholesterol reading back and they'll say it's normal, it's borderline high, it's high within that normal zone. For most of the biomarkers that we have, we can identify really a more optimized zone. And that optimized zone might be something where we say, okay, well, actually for women of my age, we find that if your fasting glucose is really between like 70 and 75, then you will probably have a lower myocardial infarction risk or, you know, a lower stroke risk. So somebody has published some paper that's looked at that and then we'll come and say, oh, okay. So actually the normal range says maybe from 70 to 80. And I'm making this up. I don't remember what they exactly are. For any of the clinicians in the audience having a freak out right now. But the idea is that if there's any, if there's a strong signal for clinically published evidence that says that we could sort of narrow that down maybe for a certain group of folks either by age or by sex, for example, then we will. And the other thing that we'll look at is we'll take large scale studies like the NHANES study, which you all are familiar with, I'm sure, But that is this beautiful registry where we have all of this biomarker data from US population that largely represents the US demographic as well, which is hard to get. And so we'll take the healthiest people from that cohort and we'll say, okay, all of you healthy people, what is your, what's the distribution of your ldl? And see if we can find that optimal zone by looking at those, those that group of individuals, and kind of take all of that, that prior knowledge, that data driven approach, and say, you know, can we really get you into this? Is there, is there a more optimal zone for you within there? So that's the first thing we'll do is we'll benchmark you and say, like, are you optimized or do you have room for improvement? But then the second piece is, if you have room for improvement, what can you do? And what can you do that doesn't require a drug? Not that drugs. I mean, I love western medicine. I love drugs. I feel like people should listen to their doctors.
- [13:35]
C
I'm on drugs right now.
- [13:37]
B
Yes, yes, great. Drugs are great. Drugs are great.
- [13:41]
A
Let's Clarify something real quick. Leon says he's on drugs right now, but he's on drugs for a pinched nerve. He's not on illicit drugs, just to be clear. Sorry to interrupt.
- [13:52]
C
No, no, thank you, Steve, for keeping me out of trouble. Wait, so Renee, hang on. I just want to make a slightly sharper point for our audience. Right. To not only are you being careful in picking out a reference frame. Right. And sort of pulling out a reference cohort, but you set a specific evidence bar where you won't recommend a lifestyle intervention unless it's been shown effective in humans across three independent clinical trials. And it moves a biomarker that's actually out of range for that person. Talk a little bit about why you pick that bar and what that bar throws out.
- [14:27]
B
Yeah, that's great. And this is where this, this is where you're probably going to have to stop me because I could go down a rabbit hole. One of the things that drives me.
- [14:36]
C
A really bad joke to interrupt it.
- [14:39]
B
Please. One of the things that you know I will fire me up. And I am scrolling social media at one o' clock in the morning is, you know, the random wellness influencer that is touting something that at worst is incredibly harmful and at best is neutral. But you know, you see a lot of cherry picking of different papers. Well, this study said, well that's cool, that's great. That that study said that MCT oil was going to lower your cholesterol because there are six others that say that it won't have any effect. You're not talking about those. Right. You're not doing a full on meta analysis of all of the literature. You're not doing an assessment of the strength of evidence from each of those papers. Was it a retrospective analysis? Was it a registry study? Was it a prospective randomized control trial? Was it a meta analysis, which is sort of the strongest analysis that we can get statistically. So I think when you're talking about making these lifestyle recommendations and of course other types of interventions as well, it's very important to look at all of the evidence. And that's, that is sort of the really nice thing. It's the reason why I actually joined InsightTracker because all of this was built before I got there. I can't take any credit for it. But the idea was that you would only make a recommendation to an individual if, if that, if that intervention and that intervention could be taking a supplement, it could be doing hiit workouts for 30 minutes twice a week. You know, something specific like that. There has to have been studies that have shown that, yes, indeed, it is going to improve your VO2 max, it is going to improve your cholesterol, it will lower your C reactive protein, and there has to be convergent evidence that says that it will do that. Not just one paper that was a big part of it.
- [16:33]
C
Yes. I mean, I think you're painting a really clear picture. In some ways, you guys are trying to solve in your own way what several reproducibility crisis reactions have been trying to solve. Right. There's a lot of underpowered study studies out there, you know, and they don't replicate. And I think that there's a lot of health influencers, I think, as you notice that take advantage of that fact and go, there's a study and, you know, there's 16 people and it's not going to. It's actually not providing as much information as you think. So your approach is really interesting. You've set the bar and we can come back to why that particular bar and where you set it is reasonable. I'm sure you've got a story about it. Let me drill down into slightly nerdier detail because, just because of our audience is a lot of informatics. So let's talk about the technology underneath. So you've shared with us that the AI underneath is not an LLM, right? And it's a. You've described this knowledge representation, reasoning, symbolic AI classic, you know, krr. Could you talk a little bit? This is for listeners who think AI means chatgpt. Right. And there are folks in our audience who probably haven't thought about it as clearly. And just think of a chatbot. What is a krr and why is it the right tool here rather than a large language model?
- [17:48]
B
Yeah, that's a great question. And I will actually say we do use large language models for certain product features, but at the core of our recommendation engine that is driven by knowledge representation and reasoning. And the reason why it's so important is because it just. It doesn't hallucinate. It is.
- [18:11]
C
Right.
- [18:12]
B
That's one of the major advantages of knowledge representation and reasoning is that it literally can't hallucinate. It's not probabilistic. It's absolutely deterministic. If you, you know, put in the same query a thousand times, you will get the same, same answer a thousand times. So. And it's fully auditable. So if you came and you were like, Renee, you told me that I should do red light therapy, and that doesn't make any sense to me, why did you give me this recommendation. I will be able to tell you exactly why we gave you that recommendation. I will be able to hand you the clinical studies that supported giving you that recommendation. And I can show you the chain of logic that said, okay, it's because you chose this goal to work on your recovery and you know, any other factor that, that could have had to do with it.
- [19:05]
C
So, you know, deterministic results that are, that are supported by transparent chain of inference rules and as well as a transparent knowledge base. So I'm going to push you a little bit into nerdy direction. Tell me if this is more technical than you want to talk about, but what's the underlying representation layer on disk? Is it an OWL representation with, you know, just tell me, talk a little bit about the cement, both the syntactic and the semantic layer of how you're storing the knowledge that you are referencing.
- [19:39]
B
Yeah, and this may be, this may be beyond my limited tech capabilities, but I mean, right now it really is just mostly in a relational database. We don't have it in any sort of graph. Like the underlying underlying data might live in a.
- [19:54]
C
It's not a graph structure. It's actually a relational database.
- [19:57]
B
The inferencing isn't necessarily done on a graph structure. The inferencing could be done on a graph structure, but we just didn't build it that way at that start. And it didn't have to be. We've done some extracting of that and putting it into a graph structure and doing some, you know, so it's not
- [20:14]
C
like OWL and swirl rules, like sort of your standard Semantic Web stuff. It's, it's a relational database and a set of deterministic inference rules.
- [20:22]
B
Exactly.
- [20:23]
C
But they're replicating that same functionality, basically, right?
- [20:26]
B
Yes, that is correct. That is, yeah. And in terms of it, you know, at Salventa, that was, that was based on, you know, the inferencing was done over a graph.
- [20:37]
C
Yeah. And it's, it's just terribly interesting. Right, because the trick with this kind of system is how do you represent the knowledge and do, you know, if you, and if you could share. Are you guys using, you know, terminology standards? Is it basically, is it snomed and loinc or have. Did you have to invent your own representational vocabulary? You know, that's, that's for storing that data in a structured way.
- [21:01]
B
Yeah. And I, I don't want to misspeak here because I don't really know the history. I think we're doing lowing mappings now, but There was a, we had to do a lot of, there is some mapping that was done, especially because we're also handling names from different labs and different countries.
- [21:23]
C
And so there's multi. Yeah. And I'm, and actually, you know, let me, let me just sort of say, give, give the conceptual question behind my question that may be easier to answer. I'm trying to, I'm, I'm trying to understand if you're going in a more standard vocabulary route or if you felt you had to invent your own. Right. So generally speaking, reusing standard vocabularies is harder. It's harder to bootstrap, but it's more scalable. Right. And it's more interoperable. So there's just a real engineering trade off. And that's sort of the question that I'm trying to, to push. I don't know if you know, because you're not an engineer, so this may be a little bit unfair to put you on a spot. I'm like, wait, you're a scientist and I'm asking you super, super nerdy stuff. So, so feel free to say I should know. We'll bring in, yeah, we'll bring in your CTO and Prohab for the next time.
- [22:12]
B
Yeah, so yes, so yeah, you should absolutely bring in our CTO and probe him on these, on these questions. And we have a, we have a product person now who's very into this. So he's, he's really pushing us for.
- [22:24]
C
So I think maybe the way to get at the same, same thing without talking technical jargon is like walk us through the MCT oil example, like where you had five studies, you had mixed results. And because it shows the reasoning engine deciding not to recommend something and talks about how the signal represents, there's not enough signal yet. I just thought that was a really interesting example from the pre.
- [22:45]
B
Meeting. Yeah, yeah. So the example here would be an analysis of, is MCT oil. Should MCT oil consumption be recommended for people with elevated ldl? So we'll go and we will scrape any, any paper that looks at MCT oil and its impact on ldl and then we will kind of extract information about that. But ultimately we look at evidence balance and you know, if there are, and I don't remember how many we currently have right now, but if we have 10 studies and three of them are positive saying that it improves LDL and two of them are negative and the remaining five are null, you know, I don't know. That's actually a borderline case. A human will make that call. But you know, we have tooling that will present it and you know, basically give you the kind of the confidence metric in how positive this would be for an intervention in terms of improving your ldl. But ultimately the call is made by a human and by actually a committee of humans. So, you know, usually there's one particular scientist that is, would be responsible for that research question and they will make a determination looking at all of the evidence and using the tools that we have to support it as much as possible. But they'll come, you know, and kind of present to the group and say like I think that there is enough evidence and this is why, or I don't think that there is enough evidence and this is why. So that's what they'll do.
- [24:28]
C
Yeah, I think that. Yeah, that's super interesting, Steve.
- [24:34]
A
Yeah. So let's unpack the, the safety layer. You've mentioned there's human in the loop several times because you know, I think having these types of recommendations that tell people to go and do X, Y or Z. I'm really glad to hear that you've kept humans involved in here. But explain to me how the safety layer tends to work. If you want to cite a specific example. I think we talked about the alpha lipoic acid and iron case. But how does all of this come together from a safety perspective?
- [25:07]
B
Yeah, so we try to automate as much as possible while continuing to keep the human in the loop. Because that like ultimately, and I should say that even though Inside Tracker is a direct to consumer company and we can't like we, we just, we need to eat our own dog food essentially. We will also like, we will offer this recommendation engine if people wanted to white box it for their own purposes. But the idea is to give them a starter pack of, of kind of knowledge and recommendations that are absolutely, that are kind of tried and true and bulletproof. And the way that we do this is even though we have automated processes to come and like scrape and do search queries for a research question, pull the papers out, pull and populate kind of a computable knowledge base with that information. The human, before anything gets committed into our knowledge base, the human has to go in and take a look at it before a recommendation is ever pushed to our product. Or we have other ways of communicating with people like via pro tips, which is not kind of this extensive as a recommendation. It's just like a little insight that is all reviewed by a human. The logic is developed by a human at this stage. I think in the future we'll kind of at least have AI propose some logic after looking at a bunch of research papers. But that's not something we've tested yet. But it's something that's like in our product pipeline to do. But all of you know, all of those kind of major steps, committing to our knowledge base, committing to our product. There's always a human in the loop there and often there is a primary human and then a backup human as well, that'll do a secondary review, particularly for anything that's going to go into product.
- [26:59]
A
So that's great to hear. Thanks for that. And then I guess the X question is talk about outcomes. Like what have you been able to show in terms of if that what your recommendations are doing are actually having a meaningful difference in people's lives? Are they living longer? Are they getting less sick? Are they having, you know, less sequelae from disease that they currently like tell you wrote a paper was published PLOS in digital health, 20,000 users was launched. Trudinal. Tell us about that. It involved fitness tracking and things like that and DNA and talk about the outcome. How does this land in a real world setting?
- [27:38]
B
Yeah, I think this is really important because one of the interesting things I think about working at a company with an established product is you've collected information over the course of a decade, so then you can look and actually see if the product is working in the way that it was intended or not. So I will caveat all of this by saying this was a retrospective analysis of completely blinded, anonymized Data from, from InsideTracker users. And it looked at users that had at least two blood draws. And the goal of this analysis was to say, all right, so for people who had, let's say, out of range LDL or out of range glucose or were sleeping poorly, if, if we identified them as a cohort at baseline, what happened to them? By their second blood draw and blood drawn forth, did they actually improve as a group and were they able to sustain those improvements? And the answer, honestly, I'm like such a Eeyore scientist, I'm like, it's nothing's going to be significant, but we should look anyway. But I was surprised that actually for the vast majority of biomarkers that we looked at that group that was out of range at baseline at a population level, their mean biomarkers values improved by their second blood draw in a statistically significant manner. And then those improvements were sustained for individuals for whom we had more than two blood draws to look at. I think one of the more exciting examples and with a smaller cohort of individuals. But for folks that actually had diabetic levels of A1C at baseline, by their fifth blood draw, they were down into pre diabetic levels of HbA1c. So HbA1c is probably our only, like, purely diagnostic biomarker that we have on the platform. But I thought that that was very interesting.
- [29:45]
A
Is your work causal? You can tell us if your work. You established causal.
- [29:51]
B
We have not. I will never say that we established causalities.
- [29:55]
C
That'd be so hard, man. An interventional trial, this would be too really expensive.
- [30:00]
A
Yeah, that's a big numbers.
- [30:02]
B
I mean, yes, it would be like my dream is to actually run that interventional trial. But yes, I mean, just from a business standpoint, nobody, nobody actually wants to run the trial. There's not a ton of money behind it. Although I'm like, what if we just applied for an sbir? I think we haven't cut those yet.
- [30:21]
A
It's not a bad idea.
- [30:22]
C
That's a good idea, actually.
- [30:23]
B
Yeah, yeah. But it's a retrospective analysis, so we can't claim causality.
- [30:29]
A
So, you know, thank you for that. I mean, this is a, I want to use the word a little fuzzy, but I don't know if that's accurate because you've done so much homework under the, under the hood and there's so much evidence under the, of what you're describing that that doesn't seem like a fair, a fair assessment. You've got so much evidence that it's, it's moving the needle. You know, you're honest with me that you don't have clean adherence data that nobody wants to give us, that it's super boring. But. So how do you reason about outcomes when you can't see whether people are actually doing the things that they've been told to do? That's a bit of a challenge, isn't is?
- [31:07]
B
And I'll give you an example. This example is in the paper of how we tried to get at this question. And just to be really clear, if you use the platform and let's just say I have chosen to have these five interventions as part of my action plan, I can go in and click that. Yes, I took my vitamin D supplement today and honestly, if I'm wearing a fitness tracker or a sleep monitor, it'll automatically check me in and say that I did or did not make my seven to nine hours a night, et cetera. So we try to make it as easy as possible for folks, and we're doing a bunch of ui Changes to try to entice people to get in and give us this data and make it easier. But yes, there are, there is, there are a couple of unicorns in there who I think deserve lifetime memberships to Inside Tracker who actually go in and track. But without that compliance information, we have to make inferences. So one thing that we can do is for, for people that we have fitness tracker data from, there are some behavioral cues we can get from that. So the example that was published in the paper actually looked at individuals. All individuals started with above optimized ldl. They didn't necessarily have clinically high levels, but above Inside Tracker optimized levels of ldl. Then we said, all right, some of you improved over time that way and some of you stayed the same or you got worse, you did not improve. So what are things that we could look at that are behavioral in nature that distinguished, we'll call them the LDL responders from the non responders. And one of the things that we saw was that the group of folks that were able to improve their LDL over the course of a year started taking more and more steps. Interesting behavior. And by the end, the people who improved their LDL were taking on average like 2,000 more steps per day, which is another mile compared to those that were not. So, you know, we try to take all the other contextual data that we have and then make, you know, make inferences. We can't claim causality again, but make inferences about what could have caused this. So there was something about that group of folks that they're taking more steps. Maybe they're taking their health more seriously. Who knows what else was going on in their life that we weren't measuring.
- [33:32]
C
Super interesting. So I want to come back to the use of LLMs in your system. And it's a mixed model. So you've got the inference engine that's deterministic and transparent. But then of course, you're using LLM to generate some of the interesting weekly summaries and those subject to typical problems of LLMs. One thing that struck me as really interesting is you drew a hard line. Recommendations must be written that they must be written by humans, not by the model. I'd kind of like you a quotable quote. Right. If you, if you want that, go use GROK or Open air, you know, OpenAI Health, where there's no human in the loop. Right. You know, we all know how to do that. So. So is this a. I guess I'm trying to understand if it's a permanent Design principle or ancurrent limitation. And the reason I'm asking is because a business model critic would say that's, that's all very nice and very, you know, humane of you, but a curated knowledge base with humans writing every recommendation just doesn't scale. Right. And the LLMs everywhere crowd will eat your lunch. They're just, they're going to out compete you on cost and production. That's a fairly sharp question. But I want to, I'm curious if you have, if you guys thought about it and what your answer is.
- [34:52]
B
Yeah, no, I think it's a great question. It's a very fair question. And so I would say from the perspective of a consumer, whether the consumer is an individual or whether the consumer is a potential business partner, our ideal customer profile includes people that do want some level of traceability in their recommendations. Right. So if you don't, then, you know, then you can go and access LLMs. Traceability has to be important. To your other point, though, it's brittle. It can be hard. We can't just scale immediately. If we wanted to add 500 more biomarkers tomorrow, it would take us some time to add 500 more biomarkers and all of their recommendations to the platform. But what we have done and done, like very careful analysis of is, you know, what is the process of. It's like, how does a bill become a law? But how do we, how do you, how do you generate a recommendation? What is the entire process that we go to and how much time does each step in the process take? What we have found is that the vast majority of time, like 85% of the time in building a recommendation was spent doing the following. Compiling all of the research papers, reviewing all of the research papers, format formatting them for import into the knowledge base, and then like doing that preliminary analysis of evidence balance. We'll call it like a, yeah, mini meta analysis. That's 85% of it. 85 to 90%, I should say. The rest of it is like actually deciding on the logic and like in writing the recommendation. So we've spent our effort in optimizing that 85% right now first. And that's been wildly, like, wildly successful. Like, we have been able to take that process down from, you know, something that might have taken, you know, an hour to 15 minutes.
- [36:59]
C
Yeah, well, so. So an obvious transition for you guys, right, given what you just described, is to move to a model where the LLM does all of that prep work and does a preliminary draft and the human is the reviewer Right. And so there's. There is the human warrant, as our friends and in design like to say, but they're, but you're saving the work. How far are you away from that is there. What capability are you currently waiting on that would unlock that next capability, that next level of your platform? Right. Is there, is there something underlying in technology that would make it easier if the LLM platforms made it available or something else in the technology stack?
- [37:43]
B
Yeah, I mean, I'll actually, I'll give you an example. So currently we are. Our first focus was in that. On that first 85% which was really compiling the papers that answer the research question, formatting them, getting them uploaded into the knowledge base and doing sort of that early preliminary analysis. But that that process has already been built and we, we use NLP for it. But then actually one of our scientists realized that it was the current NLP that we were using did a reasonable job of actually pulling out the intervention and the intervention effect from all of these papers. CLAUDE code does an amazing job of it. Right. We were like, great, actually. But when we started this process, CLAUDE code didn't exist. So now we have that. And so now we're implementing that, which is making it even faster, which is amazing. We're able to further optimize that bolus of 85%. What we are currently working on right now is upgrades to our evidence balance tools. Our. Because we do already have some tooling that allows, it allows us to do like a meta analysis on the fly. So we're upgrading that just to make that even simpler. What we haven't touched yet is recommendation writing. So that would be our next step. But really like the precise writing of the Recommendation is maybe 5 to 8% of the process. So that's why we sort of saved it to last. And to be honest with you, right. Like it does depend if, let's say a B2B customer comes in and says like, I need a bunch of allergy recommendations. We're not in the allergy space. We would have to create those de novo. And this process would be really important if the customer was already sort of in the canonical lifestyle piece. Like we have a lot of it covered and frankly, from a product standpoint, more recommendations isn't necessarily better, even for behavioral health.
- [39:48]
C
That's a good point. Yeah.
- [39:50]
B
Adoption. Yeah. Yeah. So, but it's a great question and it's a very fair question.
- [39:56]
A
So let's shift gears a little bit to the domain of AI literacy. It's a theme that's come up time and time again on our podcast, the issue being that many folks inside of big organizations actually are not. They're generally AI literate, a small, tiny percentage of folks. Your data science team didn't hire LLM specialists. Right. They became prompt engineers and LLM engineers through the course of their work. How did that upskilling actually happen in your organization?
- [40:29]
B
Yeah, I mean, honestly, we thought about hiring these sort of domain specific people that already existed out there, but frankly they didn't have a ton of experience either.
- [40:42]
C
Nobody has five years experience in how to use plot code.
- [40:46]
A
I'm sorry.
- [40:46]
B
Exactly.
- [40:47]
C
Yeah.
- [40:48]
B
And what we like, we didn't need a canonical LLM engineer. We were not building a foundation model. Right. We started differently, we wanted. So we did just end up sort of organically growing them from the team that we had. Obviously we have really talented folks who are very interested and very willing to kind of jump in and learn and start spend a lot of time researching. We do work with a consultant who has, is doing this kind of work across a number of companies, so is able to sort of bring a broader perspective beyond just what InsideTracker is doing. And that I'd say was particularly helpful. So we did have one person who I'd say did have specific domain expertise that we brought in.
- [41:36]
A
So yeah, you know, from my read where I, where I am in the industry and where I've been, it feels like something north of 97% of the folks, at least in the pharma world, at least in my last role there, really didn't have a lot of AI literacy. They just didn't understand it. There was a small group that did. They were the IT organization. Generally. It was very few of the folks that were sort of of, you know, in clinical operations or clinical development, they didn't have a, you know, to say they didn't have a clue would be, you know, accurate. They would tell me, you know, I don't even know where to begin. It's interesting that you, you know, you brought your organization forward in that space. You know, you push back that the big AI players are fueling this magical unicorn expectation that really doesn't exist. You know, I, I'm just curious as to how you guys, you know, you, you traverse that, that mountain, so to speak, and you're, you're working with it, you've got your engine running and yet the bulk of other organizations haven't been able to do that, or at least they're in the throes of it now.
- [42:48]
B
Yeah, I mean, we were really, I would say that there's two, there are two ingredients to that. And I, I would actually love to ask you where you think pharma is sitting nowadays, but I' Two magical ingredients that we had were, let's say there's a lot of employees at Inside Tracker that are just really interested in this. They're very data centered. They just, you know, we just tended to hire those folks who were just organically excited about figuring out how to incorporate AI and not even necessarily just in the tech or data science functions, you know, even, you know, across the board, ops, marketing, etc. So, you know, they were pretty tech savvy individuals to start with. But the second piece was, frankly, we just, we also got a lot of pressure from like our board, right, because our, our board is, they're out there reading all these LinkedIn articles about how AI is changing everything and you don't even need humans anymore to work at the company. And you know, they have a much more realistic, you know, idea of it than that. But you know, they, they were asking the same exact question, why, why LLMs could just eat our lunch right now. So what's the importance of a white box AI engine and how can we, you know, we really needed to show them that we were operating the business in a more efficient way because of AI.
- [44:08]
C
So yeah, super interesting. I'm going to start bringing us in for, for a landing. I'm going to ask you just a couple of questions to help us do that. So for someone evaluating consumer, health, AI product, what's the question they should ask to tell signal from the noise?
- [44:27]
B
Oh, that's a great question. I think I would actually just pose it exactly how you did, which is how does this company distinguish signal from noise? And I think if you want to be really specific, you can ask what is the evidence behind your, your recommendations and can you show it to me? Would you be able to show it to me? Would you be able to show me an example? Nobody's going to reveal their proprietary engine, but show me the example.
- [44:56]
C
So you need to, you need transparency both in what the reasoning was and what the evidence base was. And you have to have confidence that you can kind of inspect the. Both are inspectable. You're not going to necessarily stare down each one, but like it's there for reference, right. Otherwise you don't feel good. I mean, so what I like about your description is that you're, you know, and what you guys are, are showing is that this may be a fairly early prototype of the kind of neurosymbolic AI Approach where you have a reasoner and an LLM merging together like the two halves of the brain, right? So I personally don't love that metaphor. There's like two halves of the brain. System one, system two. It's probably not halves, it's cognitive modules where you need, you know, and I speak this as an old school cognitive scientist, right? So I've got my. I'm like, these are cognitive modules and you have, you sort of have a symbolic reasoning module that can handle this, and then you have a language module that's got an unexpected boost over the last, you know, five years. And all of a sudden we can plug that in and make it work with other things. But I think you guys are foreshadowing some things to come, even though the way you are bringing them together is probably not really neurosymbolic, but it's. It's starting to merge into that architecture. So that's sort of my kind of very nerdy takeaway and why the approach is interesting and why I think it's a winning move to keep going. So if you had 60 seconds with a health system or a life sciences executive in an elevator, what's the one thing about building trustworthy AI that you want to tell them to take away from how Inside Tracker was built?
- [46:31]
B
I mean, on the neurosymbolic piece, I think our chatbot is much more merged in terms of what, what exists as neurosymbolic versus the stuff that we kind of talked about before in terms of using nlp, et cetera. God, if I had an executive in. Can you repeat the question?
- [46:51]
C
Yeah, yeah, just. It's just as like one of our standard closings. So sort of like if you had 60 seconds with a health system, a life sciences exec, you're standing with them in an elevator. What's the one thing about building trustworthy AI that you want to tell them that they could take home from, you know, from, from your experience with Inside Tracker and how you guys build it?
- [47:12]
B
Honestly, I think I would probably say figure out a way to use a really robust knowledge base. And that knowledge base can be prior scientific knowledge, but that knowledge base can also be institutional knowledge that you have collected from your own data in your own proprietary, you know, studies that you've done and you've never published, but also all of the decision making that you have done over the years and all the mistakes that you have made, you know, So I think another way of doing this is like, Renee worked at InsideTracker for four years. So how cool would it be if you could actually just take what I know about what I learned at InsideTracker for four years and leave it there at InsideTracker if I left. Right. So how do we do that? I think that's an. An interesting thing to pose for larger organizations especially. Doesn't necessarily make it trustworthy. You're gonna probably need to have a human in the loop somewhere. And like, we didn't talk a lot about qa, but of course, that's a big deal as well.
- [48:15]
C
Right. But no, that's. That's really cool. So, I mean, you're. You're pointing to something that we've been exploring throughout the podcast, which is your specialization is about your data.
- [48:27]
A
Right.
- [48:28]
C
And the quality of your data and the relevance of your data and the trustworthiness of your data, as well as the transparency of the inference rules that you're using on your data. Right. If that. If decision making is really, really important. I'm kind of intrigued by, like, how do we clone Renee's brain and leave it with employer? Because to me, that actually raises some interesting IP issues. I mean, we're in a world where we are training AI to do all kinds of useful things, and the question about who owns that, I think, is unsettled and a little bit unsettling. Right. I mean, we didn't used to live in a world where me, by working as a consultant, actually leaves a piece of my brain for other people to use. And we are shifting into that world. It's an interesting world, but I think that there's some both legal, ethical, and economic questions that remain unanswered. But it was so fascinating to chat with you. Where can people find you? And plos Digital health paper that you guys recently published.
- [49:26]
B
Yeah. So I can be found as Renee Dehan on LinkedIn. Not super active on the other socials,
- [49:37]
C
but it's a toxic waste dump. Don't. Don't go there.
- [49:40]
B
Oh, I just. I have not a big face on there. Yeah, I just lurk and judge.
- [49:48]
C
That's the way to do it. That's right.
- [49:49]
B
Definitely helpful, huh?
- [49:55]
C
That's awesome. So I hope everyone will go and look for the paper and look up Renee. It was a great conversation. I just want to thank you for joining us and sharing with us your experience building and working with this interesting product. Steve, thanks as always. And I want to thank the audience for joining us every week and look forward to seeing you all again next week on Practical AI in Healthcare.
- [50:25]
A
Thank you for joining us this week on Practical AI in Healthcare. If you're ready to go beyond buzzwords and hype and explore how AI is truly transforming healthcare. Stay tuned for more conversations that get us to what works. Until next time, stay practical.