Loading summary
Transcript52 lines
- [00:00]
A
Welcome to High Impact Growth, a podcast from Dimaghi. For people committed to creating a world where everyone has access to the services they need to thrive. We bring you candid conversations with leaders across global health and development about raising the bar on what's possible with technology and human creativity. I'm Amy Vaccaro, senior Director of Marketing at Dimangi and your co host, along with Jonathan Jackson, Dimangi's CEO and co founder. Today we're moving past the hype of AI to look at a real live in the field use case that is actually working today. There's a lot of skepticism about whether generative AI can truly perform at the last mile, where languages are complex and the stakes are high. But our guests today are disproving that skepticism. Joining us today are Abraham Zarahoun from Last Mile Health and Sid Ravanutela from ID Insight. They partnered with the Ethiopian Ministry of Health to Launch Hepassist, an AI driven call center supporting 40,000 health extension workers. We're going deep into the ins and outs, the technical hurdles, the safety guardrails, and how a government can successfully lead an AI transition. If you've been wondering if AI is actually ready for the front lines, this is the episode for you. Enjoy. All right, welcome to the High Impact Growth podcast. I am so excited for our conversation today. I'm here with Jonathan Jackson, my co host. As always, John. Hey, good to see you, Avery.
- [01:26]
B
Good to see you too.
- [01:27]
A
Yeah. And we are joined by Sid Ravanutella, Chief data scientist at IDInsight, and Abraham Zarahon, Ethiopia Country Director at Last Mile Health. We are very excited to be joined by both of you, and today we're going to discuss a very specific use case of AI for healthcare workers. What happens when that healthcare worker needs an immediate answer to a complex medical question and there's no doctor nearby? So today we're looking at how, together, ID Insight and Last Mile Health have solved this exact problem with a program called Hepassist, which is a generative AI powered call center designed specifically for health extension workers in Ethiopia. So I'm really, really excited to dig into this particular story, but I'd love to start with some introductions. So, for Sid and Abraham, I'd love for you guys to introduce yourselves, share a bit of your journey into both data science and global health, and also, what drew you to this particular challenge of supporting frontline healthcare workers in Ethiopia?
- [02:32]
C
Yeah. Thank you, Amy. Thank you, Jonathan, for having us here today. Yeah, my career is a little bit of a flip flop, so I started my career in tech consulting and then gone in and out of the development sector for almost 20 years in search of, I guess, a role where I could do something that was meaningful to me and something that was technically deep and interesting for the nerd inside. And that's been surprisingly, a challenge until, I think, maybe five or 10 years ago when data science became a bigger thing. So I started my career. My undergrad thesis was a machine learning project on intrusion detection, looking at network traffic and seeing what someone's trying to hack in. And then did some tech consulting and then was bitter and jaded after that. So took some time off to work in Ghana, Papua New guinea, and then eventually Uganda, where I worked for Clinton Health, where Abraham and I both worked at Clinton Health. I think we just barely overlapped. And then after that, what I do after Uganda, went to grad school, worked for, very briefly for an education nonprofit, worked for a think tank at the Harvard Kennedy School called Center for International Development, worked in the private sector with Quantum Black, which is McKinsey's data science army. And then we've been with iDInsight for now. Coming up to six years. Next. Next month will be six years. And the way I describe it is that iDInsight is the best iteration where I've been able to find this balance between doing things that are meaningful, that I can be proud of, and doing some technically deep and interesting things. We're happy to dig into that more and share more, but I'll hand over to Ram to introduce himself.
- [04:11]
D
Thank you, Sid. I started my public health career around 20 years ago. I started working in HIV. I am part of the HIV generation, which is a very activist form of public health, very passionate about hiv. I don't work in HIV anymore, but that's how I started my career. That was a time where being HIV positive was a death sentence. Treatment was not available and countries were really trying to scale HIV prevention and treatment programs. So after spending some years working in hiv, I started working more in rural health and primary health care, more of like a comprehensive approach to delivering public health services to the last mile. And I've done that in Ethiopia and in other countries as well. I worked in Sotho for some time in very, very rural, hard to reach areas, expanding primary healthcare services in those areas in the mountains of Lesotho. And I've also worked for some time in the Caribbean, in Jamaica, working in expanding HIV services through primary healthcare. I've worked in health management as well, maternal child health, several programs, working for the Clinton Health Access Initiative. That's where we overlap with sijt. I was also part of a hospital management program through the Yale Global Health Leadership Institute. And then finally I end up, I think I had some time with philanthropy from the funding side as well, but finally came back home to Ethiopia, which is where I'm from, and started working for Lost my health.
- [05:55]
B
Wonderful. Amazing levels of experience and depth to both your careers. So really looking forward to this conversation on learning more about specific program in Ethiopia. Although obviously we want to hear how your background's led to the work in this and the exciting work that is continuing to happen in Ethiopia.
- [06:12]
A
Yeah. So I'm curious, what was the spark of or the AHA moment for this particular effort that we're going to start speaking about, which is the HEPA cyst effort, this AI powered call center in Ethiopia. Tell us where that started.
- [06:25]
D
Well, around five years ago or so, the ministry asked us to come up with a technology response which can reduce the cost of training of health access workers, which are community health workers. In Ethiopia, we call them health access workers because the cost of training was very high. The ministry wanted to digitize, reduce the cost and also improve the quality. So we came up with a solution, a blended learning, a hybrid form of learning which has a face to face and a digital component which brought down the cost of training by 40%. But the aha moment of looking for an AI solution was that even if you train competitive health workers, they might face a complicated case. You cannot train for every case and they would need support to help them manage a case, to reduce unnecessary referrals so that they could treat cases which is within their scope of practice, or to also reduce delayed referrals so that people can get treatment right away for cases which should not be handled at the committee level, should be referred right away. So training, we felt is not adequate or might not be adequate for community health workers to manage every case. So that was the AHA moment where we initially started a call center. But then we said the counseling and the support they need needs to be standardized. We can bring in AI. And that's where we started looking for technical experts and partners who can help us for this solution. And that's how we found Sid and IdeInsight great.
- [08:10]
B
And to add, I was really excited when I heard you guys were working on this project because we would deep respect for both Last Mile and ID Insights work. But one of the things that I think everybody thinks of is like, oh, does this work in local languages? Right. We keep hearing how AI models are not applicable. So Sid, as you Kind of take us through how you guys iterate on the project design. I'm particularly curious, when you got that first email, were you like, oh, this is probably going to work or oh, this is probably not going to work in general when you started brainstorming?
- [08:40]
C
Yeah, to be completely fair, we had some great ambitions when we started and then some reality checks along the way. So language is one example of it. So what we'd ideally want to do is use open weights models where we're not dependent on any provider, Google or Gemini or OpenAI. But the performance of Amharic and Oromo has been pretty mediocre in these models, the foundational models. I mean Google owns the Internet and they suck the Internet dry. So I feel like it's understandable why their model does so much better. I was even looking at some work that dimaghi, some benchmarks that dimaggi had put out and that even shows performance on a hierarchy and how much better Gemini is compared to some of the other ones. So that was one. So there's a couple of things that we would love to have and we are hoping that we, as the technology improves, we would get there. One is completely open source using only open weight models. Two is can we run these on device or at least have some capability? Some of these smaller language models are now coming out that at least some of the capability, maybe not everything is available offline, but can be increase the features that are available offline. And the third is voice in real world settings. Can we allow questions to be asked in a noisy background with kids running around and in a language not English and it still responds accurately? So these are the frontier things that we are watching carefully on and maybe we can dig into some of the work we just starting off on looking at some of these components. But when we started, this is the vision, this is the dream we want to get to. And as we evaluated various open weights models, we discovered that language support was poorer than we originally thought. There is fine tuning of course to improve that and maybe that is still a path we take in the future. But as you all know really well, that is a big investment for an NGO to take on early on in this project. But yeah, so when we started we're like, well yes, we could do this in English and online fairly easily, no problem. We've done this before. But all of these other things will require us to just stay up to date with where the technology is progressing and it is moving really fast. Things that we didn't think would be possible six months ago are suddenly up.
- [10:59]
A
Yeah, thank you. Thank you for that, Sid. I think it's helpful just to even hear those kind of the three levers that you're really keeping close eye on around language performance, open source models, the run on device and then voice. Does it actually work in the real world? And I definitely want to dig in further onto the AI specifically. But even before we go there, like, how did this partnership come together with between iDinsight, Last Mile Health and the Ethiopian Ministry of Health? How did that happen and how has that been creating this partnership?
- [11:27]
D
Yes, Last Mile Health works really closely with the Minister of Health. We have an MoU with the minister of Health, and the Ministry was actually quite excited to see the potential application of AI at the community level. We involved, I think, around 26 experts from the Ministry of Health to review to make sure that the AI that the platform that we are creating is in line with ministry guidelines and protocols. So there's a lot of stakeholder engagement and there was a lot of review processes at the expert level with the Minister of Health and the country and the government of Ethiopia is quite excited about AI. We have an AI institute. There's a lot of political will behind working on AI, so that really helped with the momentum. And then we brought in Sid and iDinsight as technical partners and we worked well together. And as Sid mentioned, the technology, some of the things that were not possible when we started, became more and more possible. The proficiency of AI on local languages, such as kind of Anoromo, became better and better now that we are actually really rigorously testing it, even implementing it as well, because the local language responses have been getting better and better and refined. We are also testing the voice to text as well. We've seen some very encouraging results, even in local languages. That's really the next frontier for us that we want to explore. So the more developments which came about, the Ministry was more and more excited and became even more invested. And I think eventually we end up winning a grant that we work together on, which helped us even strengthen the collaboration and also rolling out the pilot as well. Over to you, Sid.
- [13:21]
C
Thanks, Rabar. Yeah, maybe I'm preaching to the choir here, but community health workers are the backbone of primary healthcare in a lot of the countries that we work in. And an opportunity to even marginally improve the care provided by a single community health worker or health extension worker can lead to significant, substantial improvements in the quality of health care seen by individuals or the overall health care level for a country would improve. So focusing on these sort of frontline workers we knew is where AI can have the greatest impact. And then when we had this conversation with Abraham and his team in Ethiopia has invested in community health workers for a long time. One of the old, oldest programs on the continent. I think Abraham, you can correct me if I'm wrong. I think it's 20, 20 year old program with 40,000 health extension workers. They've invested a lot in their training, as Abraham was saying earlier, and the skills development and having that sort of a buy in political will. Well, this investment, this past investment and also this political will in improving it further is really rare. So kudos goes to the Ministry of Health for setting up such a great system and then again looking for innovation to make it better. And then last is partnering with an organization like Last Mile Health that brings a ton of context, contextual knowledge from the deep history supporting community health workers in multiple countries. That sort of a partnership is really rare. You can build AI solutions in isolation, but if you really want them to scale and add impact, you need partners like Last Mile Health. So when there was this confluence of these different factors coming together where I personally got really excited, I thought this was a rare opportunity. So yeah, that's kind of how this partnership got started.
- [15:05]
A
That's awesome. That's such a cool story and I love just hearing both of your perspectives on getting this started. And I'm curious, like it's funny, in 2026 it feels like we're starting to be in this world where every project there's like an AI element or it's we're asking questions about AI. But I'm curious for this effort, like why did you decide to use generative AI and was that, did it start with AI and let's see how AI can improve this particular program or were you looking at other solutions as well?
- [15:34]
B
And I'll build on Amy's question. How did you think about the value of learning by just trying, which there's a ton of value to just seeing how AI can help support community health workers. As you mentioned said the ministry and Last Mile just learning where AI can be applied. So as you thought about the project context for both of you, I'm curious, how did you kind of, I can see two tracks of value. It's like does the exact use case work and does it improve CHW learning? But also are we learning how we can apply AI? And I'm curious if and how those conversations weaved as the project was being designed and as we were thinking about the different potential phases.
- [16:12]
D
Yeah. So the Idea of AI came about because the problem at hand is supporting community health workers with decision making, particularly for managing a case which is quite complex. And it's very difficult to address this with a deterministic model because cases are different, each case is unique. There's different backgrounds and root causes to every case. And we felt AI is better suited to address. But something we're struggling with is that we cannot explore global knowledge, everything out there. To address the case, it has to be limited to ministry protocols, Ministry of Health guidelines. That's where I think SID recommended the RAG architecture, which limits information to Ministry of Health protocols. But it has a generative AI component. But this is like information retrieval assisted by AI. That's why it's called retrieval augmented generation. And we felt this also has a safety layer to it. It reduces hallucinations and also it would allay the fears of the Minister of Health safety issues where AI goes and provides a treatment protocol. So in short, we felt AI is better suited for this because managing a case is difficult to manage through a deterministic model. And also advances in AI were rapidly happening and we felt we could integrate language and voice capabilities which would be better suited for community workers. And we're getting very, very close to that. And the ministry is quite excited. Most of the AI solutions we have in Ethiopia are equipment related. This might be. There's now consideration of using AI assisted ultrasounds and things like that. But for a case management and decision support tool, we don't have a solution yet. So hepasys became a unique solution at the community level.
- [18:33]
C
Yeah, I can add a couple of things. So just on Amy, your question about non AI solutions. Abram mentioned earlier that what the Ministry of Health invested in is a call center where these health extension workers could call for help. So they could call in and an expert who's in front of a computer can answer these questions. But even then, the quality of response that they would get is highly variable. And of course, again, it's limited by how many people do you have available serving these health extension, you know, 40,000 health extension workers. So there is a analog solution in place to provide this sort of additional support for these health extension workers. We are just trying to supercharge them with AI. So we're not replacing any systems. In fact, we're not even creating a new system. We are augmenting an existing system that already is in place. So the step one or phase one of this project was to support these call center agents with an AI support tool that there can be another level of check as well before they share that back. They can give us feedback, they can tell us, they can filter out when AI is saying something that they don't agree with. They are the final source of truth for the health extension worker. So there was an analog solution in place and we're seeing how this could boost that before. Once we build this confidence in this phase one that the answers we're getting are good and the call center agency are giving a lot of thumbs up, then we can talk about putting this, giving this directly to health extension workers. In terms of learning. Yeah, there has been a ton. We talked a little bit about the language part of it at the start. There's also just nuances around. There's a. If you look at the documentation or the guidelines that the government has put together, there's a lot of great images in it. And visually, for example, how do you help a mother breastfeed? There's a lot of great visuals on latching and how to hold the baby. You can do that in text. But really it's really useful to pull out the right image at the right time in your conversation and send that back. So thinking about there's a lot of technical learning in terms of architectures that let you do those kind of things in terms of larger, higher level learning. We've had many conversations with Last Mile Health about this was the low hanging fruit. We know that this needs to be done. But what are other areas or other ways we can support health extension workers? Admin is one. The health extension workers go through a new village and you got to onboard every single household, which is really onerous. Can we use AI to reduce the burden on health extension workers? Admin and training that Abraham mentioned earlier as well. Can we provide better training? Just in time training, you're going to see a certain household. If we know what kind of household can we provide you with just in time training support that household. So there's all these new set of use cases that are now possible. Especially once we have a proven use case. We are showing how this, what is arguably simpler use case can be done. Well then now we are starting to look at what are these other ways that that AI can be supporting health extension workers.
- [21:36]
A
That's great. And just for my clarity, like you mentioned, the phase one where the AI is helping the call center agents, what phase are you in now is actually in the hands of the health extension workers themselves or is it still with the call center agents?
- [21:49]
D
We are in phase one at the moment. It's predominantly in the hands of call center agents. However, we have selected a small subset of health care workers and we just started to try out phase two, where is direct use by health action workers. So that has started, but most of the data we have is through the call centers.
- [22:14]
B
That's great. And when did this project start in terms of timeline?
- [22:17]
C
Was it December 2024? Is that right, Abraham?
- [22:22]
D
Yes, I think the started before that, but I think actual implementation we started in December.
- [22:28]
B
Yeah, great. So I think back then and certainly now there's huge concerns around data security. And Sid, you mentioned frontier models and being locked into performance proprietary approaches. I know it's an extremely complicated landscape and lots of countries, lots of organizations, lots of individuals are grappling with these concepts and LLMs themselves are changing so much that I think the the ground is kind of shifting underneath their feet. But I'm curious, very tactically, when you had these discussions with the ministry and how you supported them to think about their data security during testing, privacy controls, if it's were to scale, how did you think through those? I'm sure it's still evolving as all things are with AI right now. But I'm just curious, how did you overcome that? Because I think that's a huge barrier for some of our listeners and governments and funders on just how do you even get started on these things? Case data is very sensitive and so just yeah, love to hear how you overcame some of those and how you iterated with the ministry to find a path to turning this on. Because I know, I'm sure 50 people listening like yeah, I had this idea, I just couldn't figure out how to actionize it. And so it's wonderful that you have actually been able to turn it on and would love to hear more on that.
- [23:37]
C
Abraham, do you want to talk a little bit about the conversations with the government on data security, data governance?
- [23:43]
D
Yes. And Jonathan, you correctly mentioned that this is a very important and serious issue for the Ministry of Health. Well, one of the things the Ministry insists on is selecting platforms which have some sort of a commercial interest vendor looking as well. So we've tried to be as careful as possible to use open source platforms as well as for the hepasys to be LLM agnostic so that the Minister of Health does not feel like it's bound by one vendor or one tool or another. So we've tried to be as flexible as possible. There's also another very strong interest by the Ministry of Health to ensure data security and also to make sure that we have a data proclamation which Just passed, I think, a few months ago. And the government is, the Ministry of Health is very serious about where data is housed and stored as well. So we also have, we're working to reassure the Ministry that, you know, we are taking all the necessary programs, precautions, but sometimes there are some technology considerations. Availability of GPU in country is still limited, so sometimes we are forced to use AWS servers. But in the future, the Ministry is building that capability. We have Artificial Intelligence Institute which is building that capability. And in the not distant future, we hope to transition that to the Ministry and also want to build on what Sid mentioned in terms of looking at potentially offline capability, which will involve looking at maybe lighter models which won't have the requirements of storage and computing power. So with those sort of lighter models, doing this locally with Ministry infrastructure and with local protocols would become much more easier in the not too distant future.
- [25:51]
C
Yeah, these are real issues. You're totally right, Jonathan. And I don't think we have quite everything resolved. We have a path, we have a trajectory on where we want to get to. As Abraham was setting, having servers hosted in Ethiopia owned by the government. In the meantime, we are reliant on cloud service providers. They're on the continent, but AWS does not have a data center in Ethiopia. So there are some compromises, temporary hopefully, that we are making until this infrastructure is in place. One conversation that we still need to have is integration into your EMR systems. And then it starts getting into a lot of very sensitive areas. Now you have access to all their medical history, you can use that to provide richer responses, but at the same time there's a lot of privacy concerns that come into it. We're not there yet, but we are aware that these are conversations that we need to have. And what are the requirements on how we manage this sort of conversation. And it might be that we don't do EMR integration until we are able to do our own open weight models, host certain country. That might be a requirement from the government. I know that OpenAI, Gemini and all of them claim that they're not using your data for training. They're not retaining it. Actually, OpenAI doesn't say they're not returning it. They are retaining some of that data available. So there is a trust component to this as well. And I think we'll have to wait and see how this develops. In terms of lock in, we've actually been very conscious, intentional about not having lock in and by that getting technical for a second, we use an LLM gateway that allows us to Switch between models quite easily with one line of code, which sometimes is also beneficial as the cutting edge model evolves. As the new models come out, we want to be able to try new ones easily. As government policies change on which models we can and cannot use, we want to be able to switch that easily. So that sort of not having any sort of a locking has been part of the design from day one.
- [27:47]
B
That's great and makes a ton of sense. I think the need to kind of think through a future where the open weight models are good enough or very good, not just good enough at certain use cases is definitely how we're recommending people think about these conversations now. So even if you start with one of the frontier models, plan for a world where you can migrate off, possibly it's even cheaper in terms of the run cost over time as well. So that's great. So we were talking about the data security piece, but also the cost piece I just mentioned, and I'm very curious how you've thought about this, modeled it. Even cents per interaction can add up to a lot if you're talking about an entire national CHW workforce. So how have the cost discussions been modeled? Is that something that's kind of too early to be thinking about? Because the tech is changing a lot. But how have you had those discussions and thought about scalability and costing from that perspective?
- [28:38]
D
Yes, I think the costing discussion conversation is ongoing with previous technology interventions. What we have seen is that the development cost might be high, but when you scale it, the per unit cost comes down. As you scale, and with AI, with things changing by the minute, we hope that the costs are coming down, but there's a lot of costs. The opportunity costs that we need to consider as well. If a person is treated with quality treatment at the community level, you are reducing cost of transportation, you're reducing cost of care at higher level. So there's a lot of even hidden costs which are associated with lack of quality of service, which is unnecessary referral or delayed referral. If a referral is delayed, it means you might need to go to the hospital. You might. So there's a lot of preventive costs which you gain from using AI or improving quality of service, which we need to measure because I feel like directly measuring only the developer costs might send the wrong picture. So as we get more data, we'll be able to assess. Just to give you one data point, up to now there's been 6,555 AI supported consultations, and out of those consultations, 53% of them have been treated at the community level, 37 have been referred. So this is not a small number. It really tells you that in a short amount of time, more than 6,000 AI supported consultation, more than 6,000 cases being addressed is. We are very, very encouraged by that. But if you assume the cost of even some portion, sizable portion of cases, if they were misdiagnosed or not treated on time, or they were referred while they were not supposed to, the cost implications are huge, which we need to measure.
- [30:57]
B
So I love that description of how you're thinking about roi. Clinton Health obviously did a ton of financial modeling during promotion of HIV procurement. I'm extremely curious for governments listening, for other partners listening, that question of cost avoidance I think is often very difficult to model and very difficult to kind of win an argument on. How do you think about that? Both practically right now, but also just given your history of working in the HIV space where this was a critical issue to talk about, whether it was first line to second line regimen changes or other externalities and societal issues. So with AI, if I can avoid a bad treatment outcome or the wrong treatment outcome, how do I kind of think about the ROI of that? I'd love to hear your opinion on that.
- [31:44]
D
Yeah, it's quite a bit complicated. Sometimes we have this need to quantify each and every cost component, which is sometimes a bit difficult to do, but I think it can be done. It definitely can be done. It's not only related with patient outcomes. There are logistics and administrative costs as well. I just gave you the number of the consultations, but the number of calls is much, much bigger. We have 18,000 calls that health Access workers made to call centers, which shows you that the demand is very, very high. We are estimating that on average health extra workers have eight course every two months, which is around. It can be four course a month, which is four cases a month, which is very high. If you Multiply IT by 4. 40,000 COMTA health workers throughout the country. It's a massive number. So the demand for case management is there. Even with these call centers. The number of calls which have been converted into consultations is only 6,000, which means that there are calls which are not answered, calls which are rejected because these call centers are health center workers who do not have the time sometimes, or who might be managing other cases. Which really makes the justification for pushing to phase two, putting this AI devices in the hands of COMPTIA workers. So when that happens, the costs of, especially at scale, the cost of AI becomes more and more justifiable and as the cost of technology and the cost of AI reduces as well, we feel like this is going to be more and more affordable.
- [33:38]
A
Yeah, those numbers are really fascinating. And I imagine that four calls a month will grow as health extension workers realize just how valuable the services. Right. Like I find for myself with my own just chatgpt usage questions I used to have that I would just be like, I'll never be able to figure that out. I'm like, actually I can just ask AI this question and get a pretty interesting answer and help evolve my thinking. So I, I am curious, like, sorry, go ahead.
- [34:03]
C
No, that's 100% right. Once you reduce the barrier to asking questions, you don't have to actually call someone, wait on the line, speak to a human, but just while doing something else, ask on the side. I expect the volumes to be even higher than the calls that are coming in right now. Can I just say one point on costs? We used to work in tech or build technology where you pay this huge fixed cost and you're like, okay, marginal cost for bringing on a user is practically zero. So let's scale this thing. Now. Every person you bring on or every new user you paying a non zero variable cost. Right? So the cost model has changed substantially. When I make this argument, the counter argument often presented is that hey, token costs are going down. Don't worry about it, the future will be fine. Yeah, but newer models are coming out which are still. You want to use the latest model, not the one that came out last year. And second, there's now thinking models and new methods of doing guardrails and LLM. As judge, the token cost is going down, but the number of tokens you're using is also increasing over time. And then I feel like we're still at. It's not in my head the equilibrium is staying the same. Token cost going down, but demand for it is going up and it is a thing. And maybe hosting your own models is the answer, but then you need a certain volume scale for that to make sense as well. You're paying for GPUs, the computer's not cheap. It doesn't make sense to do that for 10 users. It makes sense for hundreds of thousands of users. So you need to get to that scale so you can start hosting them. But yeah, the cost is like a, is an ongoing thing that I'm always constantly watching for. What is that switch point when you're like, okay, no longer using, calling an API, time to host your own model. And the trends on where this is going. But yeah, but I like Abrams approach as well and like costing it out as a whole. Not just like, hey, here's the AI component. How much does it cost?
- [35:50]
A
Yeah. So I want to shift us a little bit. I think we've dug in a bit on the kind of tech side, the government considerations, things like data security, cost, et cetera. And I want to sort of bring us to the user side of the equation and something that Dimaghi we speak about a lot is what we call design under the mango tree.
- [36:10]
C
Right.
- [36:10]
A
Which is designing products with our users. I'm curious just to hear a bit about how did you. How have you incorporated users in this design process? What kind of feedback have you been getting? What are some of the barriers you're seeing? Are there things that have changed in the offering based on what you've learned from users? Yeah. Love to hear from either of you.
- [36:29]
D
As much as possible. We've tried to be user centered. We've had several consultations with Ministry of Health, with nurses and midwives who are the core agents, and also Health Extra workers as well. And we've tried as much as possible to iterate different versions. There's several functionalities that we have included which have continuously improved the tool. This can be trying to get user feedback, allowing for call agents to rate the responses from HIP Assist. This can be including maybe broader documents, reference documents, because how it works is that you upload the guidelines and modules where you want the AI tool to use. So we have included reference documents for so that there's some sort of a bigger body of knowledge. But still within the ministry guidelines, we have included citations where the AI needs to reference where it's getting the responses from. Okay, this is how you should manage this child. And this response was given to you from so and so guideline, page 70, page 40, so specific references so that they could go verify those guidelines as well. And the latest addition has been introducing a character called Hawa. This is one of the first health workers we used in our user center design process. So we named this virtual assistant after her. So Hawa is going to be like Alexa or Siri, which is embedded with HEP Assist. Hawa is also a character we have used for our blended learning. She is the best trained health extra worker in those video stories. So we felt that let's take Health Extra workers through that journey where they're used to watching animated videos for their blended learning to learn about, be it maternal health or newborn health. And we integrated now Hawa into the HIP assist platform where they are having a conversation with Hawa and Hawa makes the, the experience more human and less machine oriented. So lots of user feedback be incorporated to improve the tool and definitely continue to do so. So especially now that we are trying direct use by health workers, we feel like we will get much more feedback which we are committed to improving to as much as possible.
- [39:14]
C
Yeah, just on a couple of technical notes on this. So there's this, as Avraham said, there's like the intensive where we, you know, sitting with these end users and watching them and getting feedback from them and this is all in person and doing multiple cycles of this. And then there's the softer touch which is within the app itself. They could give qual and quant feedback. They could give us a thumbs up and say I like this, I don't like this and also why they didn't like it. And then we can analyze that and of course using all our traces that we, we keep track of, we can do, we do error analysis as well. So they give us a thumbs down. We could go in and kind of analyze where and looking at our traces where in the chain in our pipeline, was there a failure? Was it because we didn't have to document? Was it a translation thing? What did it not take some context into account? And that allows us to continuously improve. And we've made a lot of changes. As Abraham said, one is like adding this supplemental documentation and how do we use that along with primary documentation to answer certain questions. And then there's. Over time we've also built an evaluation set that allows us to do this continuously as we are making changes to a solution to see how it's performing on this evaluation side. Are we getting better with every change?
- [40:27]
B
So we just have a few minutes left and I'd love to hear. Sid, you mentioned evaluations. This is something that I've spoken a lot about with our Open Chat Studio platform and scenario we're invested in because it's just so hard to tell as you continue to improve things in exactly which ways did it improve? How confident are you in that improvement? So that's one nugget that I was really excited to hear as part of the project. But I'm curious for our listeners who are all trying to deploy AI use cases at this point. I would guess you started this in late 2024. You've had all of last year to kind of see these tools explode in their capabilities, see models get much more competent. But if you were starting a project today or giving advice to somebody starting a project today. Do you have your top three learnings or design aspects or just how to think about problems like this for those, I wouldn't say getting started because everybody's already gotten started. But just as you're thinking about these types of projects, like you guys have such a wealth of experience, practically really turning something on, what would you say? Or things you wish you knew at the start or things you learned that were critical? And same question for you, Abraham.
- [41:30]
C
Yeah, this is a good question. Let me see if I can construct something on the fly. So one advice that I give to most organizations, almost all organizations, is to start with off the shelf tools. Partly because if you haven't built anything or this is a new thing that you're adding, you need the experience to articulate what exactly you want. It might be really hard to do that when looking at a blank page, but it's easy to react to something. So there's a lot of off the shelf tools now available that can augment AI into your process. They might not be perfect for you, but that's what I recommend that most people start with and then that'll help you say, ah, actually I want feature A that is missing on this and that's really important to me and it's worth me investing in building something on my own. That's one thing that we have learned. What else? I think evaluation is something that you start from day one. I think even on this project we left that a little bit later. And it doesn't have to be like a massive data set. You can start with 50 use case or 50 conversations, something really small, something realistic, just to let you know that you're trending in the right direction. Temina, who leads the agency fund, once said that evaluation is not separate, it's part of product development. And I completely agree. I think that is something that I wish we had internalized a lot more. And now when we speak to organizations, that's one advice I give them as well. Let me pause there, see if Abraham has any ideas while I think of a third one.
- [42:54]
D
Well, what I have seen in our field, in the public health world, especially at the community level, or other solutions as well, there is hyper enthusiasm on one side of the spectrum and there is so much skepticism on the other side. My advice I would to give is try to maintain that balance. I feel like on the hyper enthusiasm side there might be a tendency to regard AI as the new solution for everything, which may not necessarily be true as there are solutions which are best addressed by technology through the good old deterministic models that we've been working on. So jumping on AI for the sake of AI or just because it's fashionable or just because we are super excited and enthusiastic about it may not be the right way to go. So is it really a problem we need to solve through AI is I think the number one question which needs to be answered. On the other side of the spectrum, I feel like there's so much skepticism on AI. There's a lot of people who tend to shoot it down saying that this is a rural area, this is the last mile, AI is not an appropriate solution. And I feel like we have disproved that. I was in a gathering recently where we're trying to come up with standards for training of community health workers and I was proposing community health workers to be trained in AI as part of the pre service training because chances are they will be exposed to AI. I saw a lot of naysayers in the room and I think there is a lot of skepticism. I feel like we need to produce data to convince that skepticism because we should not let the AI revolution pass us. And there's a lot of potential and there's a lot of problems we could solve through AI as well. So I feel like the only way to do that is to produce the data and show that from a programmatic point of view, from quality of service, from patient outcomes, even from costs and return and investment perspective, AI has of a lot, lot of potential. That would be my advice.
- [45:25]
B
Yeah.
- [45:26]
C
Can I just add one thing to
- [45:27]
D
Abraham just reminded me.
- [45:29]
C
Yeah. That what the point that Abraham was making early on is exactly right. One of the things that I encourage organizations is to think about direction more than speed. I feel like when you're starting off and you're coming up with the AI use case, identifying the real problem that you're trying to solve, is this truly a barrier to you achieving your mission? Is AI really the thing that's going to solve it? Often of people are really excited about they started fast, but instead spending some time on getting that direction right before you put the, I don't know, your foot on the pedal. But yeah, that's one last advice for organizations.
- [45:59]
B
That's great. And I think one thing I was reflecting as I was hearing both your expert suggestions was if we were doing this 15 years ago, we could have made most of those sentences replacing AI with like digital or mobile apps or those things. So I think history doesn't repeat itself, but it rhymes. And I think a lot of what we're hearing is there's so much potential. CHWs are always more capable than people give them credit for. Technology can have a huge impact and there's plenty of ways to use it. Totally wrong and it's not going to work for everything. So I think that all is just really rhyming with stuff I was saying 10 or 15 years ago on digital in general. But we really appreciate your time and so wonderful to hear about this project. We'll drop a ton of links in the show notes also to some of the things we talked about. Amy, back to you.
- [46:42]
A
Yeah, thank you so much Sid and Abraham and Abraham, I think one of the things you mentioned there at the end is like you're disproving a lot of the skepticism around AI for last mile use cases. And so I'm just so grateful that you both showed up today to kind of take us through a bit of your journey. And like John said, we'll include some links to materials so if folks want to dig deeper into what you're working on. Thank you both.
- [47:03]
B
And Amy, just a great point. I do just want to like really hone in on the fact that this is a live thousands of calls getting support with frontline workers in an African language. So everybody who has an objection to any one of those things not being possible, this project proves it is and it's really exciting to hear about that.
- [47:22]
D
Thank you both. Always energizing to talk about our projects and yeah, well thank you.
- [47:27]
C
Thank you for having us.
- [47:29]
A
Thank you so much.
- [47:30]
B
Thanks everyone.
- [47:31]
A
We want to extend a massive thank you to Abraham and Sid for sharing such a grounded, practical look at how AI is actually working in the field today. It's one thing to talk about the potential of these models, but hearing about thousands of live AI supported consultations in Ethiopia is truly exciting. Sharing a few takeaways for others considering AI in your efforts. First, augment don't replace Hepasist succeeds by supercharging existing human LED call centers rather than trying to automate people out of the loop. Second, the power of Personas to make the technology feel less machine oriented and more human. The team created a virtual assistant named Hawa. Hawa serves as a familiar expert colleague rather than just an anonymous chatbot. Third, future proof with LLM gateways, Sid emphasized the importance of staying model agnostic. By using an LLM gateway, they can switch between different AI models with a single line of code, allowing them to adapt as local language perform, performance improves or policies change. Fourth, ROI is more than cents per token. When discussing costs, Abraham argued for a broader view of return on investment. He noted that we must measure the hidden savings, like the reduced transportation costs for families and the avoidance of expensive hospital stays that occur when a case is successfully managed at the community level rather than being unnecessarily referred. That's our show. Please like rate, review, subscribe and share this episode if you found it useful. It really helps us grow our impact and write to us@podcastemangi.com with any ideas, comments or feedback. This show is executive produced by myself. Pradhana Balachander and Michelle Abalencia are our editors, Natalia Glowacki is our producer and cover art is by sudanchukanth. A final note, in the spirit of transparency, we use AI to assist with guest research, copywriting and post production so a small team can produce a high quality show. All AI assisted content is reviewed and edited by humans and we retain full responsibility for what you hear.