Special episode: Can AI safely support mental health?

Summary

---

Podcast Summary

This Week in Global Development: Special Episode – Can AI Safely Support Mental Health?

Date: November 19, 2025

Host: Katherine Chaney (Senior Editor, Devex)
Guests:

• Professor Bilal Matin (Chief AI Officer at PATH)
• Kristelna Reynicke (COO, South African Health Products Regulatory Authority, SAPRA)
• Professor Miranda Wolpert (Director of Mental Health, Wellcome Trust)

---

Episode Overview

This episode explores the growing role of generative AI in closing the global mental health care gap. With over a billion people affected by mental health conditions worldwide and a chronic shortage of clinicians—especially in low- and middle-income countries—AI is positioned as a promising tool to extend care. Yet, the conversation delves deeply into concerns around safety, oversight, and the unique regulatory challenges AI poses, particularly the tension between innovation and risk, and the need for locally led regulatory frameworks. Central to the discussion is CARE mh (Comprehensive AI Regulation and Evaluation for Mental Health), a new initiative positioning South Africa as a leader in ethical AI governance.

---

Key Discussion Points and Insights

The Scale of the Mental Health Challenge

• Global Need: Over 1 billion people are affected by mental health conditions (00:03).
• Workforce Gap: On average, only 13 mental health workers per 100,000 people globally, with the worst shortages in low- and middle-income countries (00:15).

The Promise—and Perils—of AI in Mental Health

• AI Can Expand Access: Generative AI can “completely transform treatments as we know them,” offering access, language translation, and multimodal interaction beyond just chatbots (Miranda, 07:20–08:25).
• Risks of Harm: Without robust oversight, AI solutions may cause harm, especially when tools claim to provide treatment without validation (Bilal, 02:11–05:07; Kristelna, 05:30–06:43).
• Blurring Lines: Many mental health apps sit in a grey zone between “well-being tools” and genuine medical devices, with varying regulatory scrutiny (Bilal, 02:11–05:07).

Regulatory Innovation: The CARE mh Initiative

• Policy Incubator: CARE mh is a regulatory policy incubator launched at the G20 Social Summit, funded by Wellcome Trust, run by SAPRA, and supported by PATH (00:55–01:15).
• South Africa’s Leadership: Positioning Africa not just as a recipient but as a developer of regulatory standards for AI in health (Kristelna, 06:00–06:43; 23:16–24:36).

The “Sandbox” Approach to Regulation

• Definition: Regulatory sandboxes provide a safe, collaborative space for regulators, innovators, researchers, and people with lived experience to co-design, test, and iterate solutions (Bilal, 21:23–22:30; Miranda, 19:14–20:50).
• Advantages:
  • Real-time learning and feedback
  • Clarity for innovators
  • Continuous postmarket monitoring (otherwise known as postmarket surveillance) (Bilal, 11:18–13:07)
• Quote: “It’s not a game anymore… The world is too complicated for that at present and… we don’t have all of the right answers.” —Bilal (21:23)

Regulation as an Enabler, Not a Barrier

• Clarity Over Burden: Lack of clear, predictable regulatory pathways is a greater barrier to innovation than the regulations themselves (Kristelna, 17:02–18:24).
• Guiding, Not Policing: SAPRA's aim is to set clear rules so innovators know “how to win the game”—not to stifle them (Kristelna, 17:02–18:24).
• Quote: “If you chase 22 people on a football pitch and say play the game without rules, you have chaos. But once the rules are clear, people can enjoy and win.” —Kristelna (17:40)

Importance of Local Regulatory Ownership

• Cultural Context: AI tools validated elsewhere may fail in South Africa’s highly diverse environment—with 11 languages and multiple cultural norms (Kristelna, 23:16–24:36).
• Accountability: “We cannot outsource that responsibility to other regulators who have actually not assessed [the] tool within our context.” —Kristelna (24:12)
• Trust: Loss of trust due to local harms can derail progress (Bilal, 24:48–27:03)

Global Shift: Regulation Shaped in the South

• Broader Trend: Innovation and regulation are now increasingly being driven by African actors, not just imported (Miranda, 27:30–28:13).
• Quote: “We have an opportunity here to create a world of mental health that’s different from the world that’s currently in the US or the UK… with a different set of actors.” —Miranda (27:48)

Barriers Beyond Regulation

• Payment Models: Health economic considerations—determining which innovations are affordable and worth adopting—are critical (Bilal, 28:42–29:33).
• Investment: Additional funding and focus are needed to scale mental health innovations (Miranda, 29:33–30:24).
• Infrastructure and Data: Reliable connectivity, robust data systems, and a workforce trained to use and evaluate AI are all vital, especially in Africa (Kristelna, 30:33–31:43).

The Five-Year Vision for Ethical AI in Mental Health

• Personalization and Choice: Mental health support will look “completely different,” with people empowered to choose safe solutions that fit their needs (Miranda, 32:18).
• Regional Leadership: Success would mean approvals by SAPRA are used as a reference point by other African countries—moving the “locus of control” to the continent (Bilal, 32:44).
• Predictable Rules and Capacities: Regulators will have more mature frameworks, better monitoring, and innovators will have clearer standards at regional and global levels (Kristelna, 33:21).

---

Memorable Quotes & Timestamps

• On Regulation and Innovation:
  “If you chase 22 people on a football pitch and say play the game without rules, you have chaos. But once the rules are clear, people can enjoy and win.”
  — Kristelna Reynicke (17:40)

• On the Promise and Risk:
  “There is both great opportunity, like you said, but also the potential for things to go very, very wrong… We should probably figure this out sooner rather than later.”
  — Bilal Matin (04:48)

• On the Need for Local Regulation:
  “We cannot outsource that responsibility to other regulators who have actually not assessed [the] tool within our context.”
  — Kristelna Reynicke (24:12)

• On Trust:
  “It only takes a few people to get hurt for us to lose the trust of all of the individuals in that ecosystem who need help.”
  — Bilal Matin (26:00)

• The Vision:
  “My prediction is that within five years we will see mental health support in ways that none of us are envisaging now...”
  — Miranda Wolpert (32:18)

---

Key Timestamps

• 00:03–01:15 — Framing the mental health care gap, AI’s promise, and introduction of CARE mh.
• 02:11–05:07 — Bilal Matin on the landscape of AI health devices and regulatory ambiguity.
• 05:30–06:43 — Kristelna Reynicke on regulatory gaps and risks to vulnerable populations.
• 07:20–08:25 — Miranda Wolpert on transformative AI opportunities beyond chatbots.
• 11:18–13:07 — Bilal Matin on the need for regulatory “sandboxes” and collaboration.
• 17:02–18:24 — Kristelna on how clarity can enable, not stifle, innovation.
• 19:14–20:50 — Miranda on why Wellcome funds regulatory capacity and sandbox initiatives.
• 21:23–22:30 — Bilal further explains the sandbox concept.
• 23:16–24:36 — Kristelna on the necessity of local regulatory ownership.
• 27:30–28:13 — Miranda on Africa’s regulatory and innovation leadership.
• 28:42–31:43 — Panel reflects on other scaling barriers: payment, investment, infrastructure, data, capacity.
• 32:18–33:21 — Five-year vision for ethical, scalable, locally driven AI in mental health.

---

Conclusion

This episode highlights both the urgency and complexity of harnessing AI for mental health globally. The CARE mh initiative in South Africa is presented as a model for how forward-leaning regulatory frameworks—developed and led locally—can both spur innovation and ensure patient safety. The core message: AI in global mental health must balance its enormous potential for good against considerable risks, with local context and clear guidance at the heart of responsible progress.

Transcript

[A]

Foreign.

[B]

Hi, everyone. I'm Katherine Chaney, senior editor for special coverage at devex. Today we're exploring whether generative AI can help close the world's mental health gap and what it will take to ensure it's used safely and responsibly. Mental health conditions affect more than a billion people worldwide, making them one of the biggest causes of disability, especially among children and young people. Yet on average, there are only about 13 mental health workers for every 100,000 people globally, with the deepest shortages in low and middle income countries. AI offers real promise to extend care to millions who currently go without support. But the risks are just as real. We've already seen chatbots cross into treatment claims, blur regulatory lines, and in some tragic cases, cause harm. In this conversation, we'll explore both sides, the promise of AI to expand access and the regulatory backbone needed to deliver on that promise without making patients shoulder the risk. This episode is sponsored by path, a global health innovator turning bold ideas into real world impact so people everywhere can live healthier, longer lives. Joining me are three leaders helping to shape the future of AI and mental health. Professor Bililal Matim, Chief AI Officer at PATH. Kristelna Reynicke, Sarah, COO of South African Health Products Regulatory Authority, or SOPRA, and Professor Miranda Wolpert, Director of Mental Health at the Global Charitable Foundation. Welcome, Bilal, Kristelna, Miranda, thanks so much for joining us. So I want to hear from each of you about CARE mh, which stands for Comprehensive AI Regulation and Evaluation for Mental Health. So this initiative launched at the G20 Social Summit, and it's really positioning South Africa as a global leader in ethical AI governance. It's a regulatory policy incubator funded by Wellcome Trust, run by SOPRA and supported by path. But first, I want to start with the need for an initiative like this. So I want to start with you, Valal. You now sit on the product development side, but you were previously a funder. So what are you seeing in the landscape right now, especially when it comes to tools that are really blurring the lines between support and treatment.

[A]

Thanks, Katherine. And let me take a step back and just, just to provide some context about how many AI enabled medical devices there are. If you look at the FDA's website, there are now over 1200 that are registered in the United States. The UK doesn't have as comprehensive a register, but many, many there. And the reality of the situation is few, if any, are mental health specific. Some will deal with issues that are experienced by individuals experiencing mental health conditions. But Few and far between. Right, and that's despite the relative scale of the burden that you've just articulated. I unfortunately can't even tell you how many there are in sub Saharan Africa because there's only one regulator that's currently systematically registering these technologies. But I'm hoping that's going to change pretty rapidly with Care MH and the work that we're going to be doing over the next few years. It's not that the mental health AI solutions don't exist and it's not that they aren't being used. They just tend to fall at this awkward intersection of what's considered either a wellbeing solution or a true medical device. And the thing that distinguishes those two, it really boils down to two criteria, its functionality and its intended purpose or the use of the tool. Functionality is an easier one to explain. Right. If you can replicate the calculation being done by a tool using a piece of pen, a pen and a piece of paper, you're fine. That's low functionality. You don't really need to be concerned about it as a medical device. High functionality is pretty much everything else. And in the world of generative AI, you can pretty much treat that all as high functionality. Where it gets slightly more complicated is the intended purpose, use stuff. And the way to think about that is if you're dealing with something that could be related to a mental health condition, but is you're experiencing it at a subclinical level. So lots of people have poor sleep and could benefit from using the digital therapeutics that are out there that help you get better sleep. That's not necessarily a medical device. If you're experiencing poor sleep in the context of having depression, where it's one of the key symptoms that people can experience, at that point, it can tip over into being a medical device. And the challenges that because this has been so poorly understood for so long, there are tens of thousands of apps on app stores which are just, let's be honest, fall short of expectations, they don't ride that line correctly. And there are brilliant academics that have documented this problem, like John Torres at Harvard. The real urgency that we're experiencing right now is that the gen AI, the generative AI, the large language model based solutions, specifically the chatbots that can do everything from triage to diagnosis to potentially even providing therapy, suddenly change the risk paradigm, right? There is both great opportunity, like you said, but also the potential for things to go very, very wrong. And so there's this renewed sense of, oh, we should probably figure this out sooner rather than later.

[B]

Absolutely. I don't want to bring Selna into this because Bilal, you mentioned that the capabilities of these chat bots are really changing the risk paradigm. And I do think that brings a lot of urgency to our conversation. So, Kostelna, I want to hear from you, bringing the regulatory perspective. When you see these tools entering the market so quickly, what worries you most about the current lack of oversight?

[C]

Yeah, so. So it is an interesting question. And I mean, I think the face of AI tools that is currently entering the mental health space, it's quite honestly moving a lot faster than the necessary oversight. And as Bilal says, many of these products now blur the lines between wellbeing, support and treatment. But there isn't actually adequate evidence for this. And the biggest concern, I think, from a regulatory perspective is the potential harm that it may have the vulnerable populations when these claims are made without actually the proper validation or when the tools offer guidance that has not been clinically assessed yet. So you've mentioned when you opened the discussion that there is also a global equity risk. So if African countries rely solely on foreign approvals, we do risk importing tools that's not really suited to our languages, our cultural context, or even the clinical pathways that exist locally. And that's also for the regulator for sapra, why these guardrails are essential to protect, protect the public and also maintain trust in these fantastic innovations that we see coming through and also enabling greater access to mundo.

[B]

Yeah, it's a great point. And we'll dig further into this that we often hear we need guardrails in place, but you're saying actually who develops the guardrails really matters. Miranda, I want to go to you and for kind of the why this matters from an optimistic point of view, assuming we can get the right guardrails in place. I'd love to hear from you as what you see as some of the most exciting opportunities on the horizon for AI and mental health.

[D]

So I think, as Bilal said, we have a huge level of need and a very small workforce, and generative AI and digital in general has a potential to transform access. But it's actually even more than that. It has the potential to completely transform treatments as we know them. So I know people talk about chatbots a lot, but there are also all sorts of other things that generative AI can do that could transform treatments, not just mimicking a human interaction. So it can use multimodal data to allow people to understand more what's going on. It can allow translation between different languages in A much more easygoing way. And it can allow voice use and voice recognition that will address people with lower literacy rates in terms of increasing access. So there are many things it can do, and there are even new treatments developed that are going to be completely different from the model of a person sitting in a room chatting in the future. So I think this is part of a sort of revolution that's happening. And what we want to make sure is that revolution balances innovation with safety, and that we have regulation that's fit for purpose, that can move at the speed the revolution is moving at, but also protect people from potential harms.

[B]

So we often hear when it comes to emerging technologies like AI, that there's both the greatest opportunity and the greatest risk in these low resource settings where, you know, they're very vulnerable populations. They have a lot to gain, but also a lot to lose. Miranda, I really appreciate that you brought in the lens for us beyond chatbots. I think often when we think of generative AI and mental health, we think chatbots. You pretty much provided a few examples beyond that. But I wonder if any of you can paint the picture even more concretely. Like in some of these most vulnerable contexts where we have major health worker shortages, mounting mental health needs. I would think what is an example where, you know, again, with the right guardrails in place and it really matters who develops those guardrails, we could have some AI tools that could truly be game changing for these populations. I'd love to hear whether it's in a health setting, an education setting. I know we were talking quite a bit about young people, specifically, what could this look like?

[D]

And some of the exciting projects that those teams are looking at at the moment include things like working on youth networks that already exist, that are using digital data already to see how we can use generative AI to help support those conversations, to become more therapeutic. Working with mothers who have depression following the birth of a child to get early intervention that's tailored for their needs based on the input that's coming in through their voice, through their messages, using the tools to translate existing measures of mental health. So those measures are right in the languages that people are using on the ground. Those are just some of the examples of the things that we're seeing coming through. And this is very much at the sort of cutting edge of development. There are other more sort of traditional things that are already happening. For example, people are using these tools to train up people who are on support lines, to help people in crisis. You can train with these tools. Enormous numbers of people. You can reduce the level of burden on existing staff by really triaging and by creating note taking in ways that have never been done before. So there is a sort of revolution at all stages in the care pathway, both in the health service and direct to young people and in schools and communities.

[B]

Also wanted to ask a question about the timeline here. When it comes to where regulation comes in, you know, how are these tools being developed in such a way when they're really being developed faster than regulators can respond in such a way where innovators, funders and regulators can collaborate before products reach patients? On the one hand, feedback is important, but on the other hand, again, we don't want the patients to shoulder that risk. So what are you seeing emerge?

[D]

I wonder if this is one for Bilox, I think, and for Christiana, because this is where the sandbox comes in.

[A]

Kevin, I think you've set us up perfectly for talking a little bit more about Care MH and really why it exists. Because the criticism has always been that industry doesn't have enough of a voice to share the challenges they're experiencing, that regulators aren't responsive enough. And that clarity arrives far too late for a field that is moving almost at breakneck pace. And really, when you bring those all together and you try and solve them, it's where you, you fall over the idea of a sandbox, right? How do you bring together everyone that has a vested interest in trying to solve the problem and figure out the solutions in real time learning? By doing so, we're privileged to be able to partner with Kristelna and the Safra team to say, okay, what does the ecosystem of AI enabled medical devices for mental health and more broadly in South Africa and globally look like? How do we bring together a Vanguard cohort of several of these products? How do we then sit down with them, understand where they are today, where we think the challenges are based on our current understanding of regulation and the open questions set them challenges to come back to us over the course of six, nine months with new evidence that suggests that they are safe and effective in this context, or new solutions for real time monitoring of the outputs. Because it's not just about that upfront assessment, it's also the continuous monitoring that needs to happen, which is called post market surveillance for these products when they're being used by individuals out in the community. How do we figure that out together so that we can begin to advance these products towards market authorization? Because what we really care about is getting them to the bedside, getting them into people's hands because they're not being used, they're really not meaningful as solutions to the both the healthcare workforce gap that you described, but also the burden of unmet need.

[B]

I want to ask you a quick follow up to that, Bilal, because I feel like this is something I hear across the tech for development landscape. There's this tension between some innovators saying that regulation can stifle innovation. But then there's also the argument that good regulation can enable good innovation. And I think you come more from that side of the argument. So can you unpack that a little bit and how you see care MH as this example of good regulation enabling good innovation?

[A]

I'm such a true believer, Katherine, that I wrote what I described as a love letter to the concept of regulation of AI in healthcare, which I think only a true nerd would ever have done. And really it comes from this belief that if you speak to innovators, they will tell you that the single greatest barrier that they experience on the regulatory side is a lack of clarity. Right? That uncertainty introduces risk for them because they then need to figure out what the regulator might have meant. And either they spent too much going further than they needed to, which is not a wise investment when you're thinking about early stage innovation. That's venture capital funds that they probably didn't have to waste. Or the other side is that they take an undue risk and they end up on the wrong side of the law and someone gets hurt as a consequence. Right? That's what we're worried about. My argument here is that if we can build it quickly enough to meet the needs of the moment, we will help these innovators make better decisions. And I'm not worried about the large scale medical device manufacturers that operate in 50 to 100 nations. They have legal departments large enough to help them navigate this problem and figure out what is the risk based decision that they want to make. I'm worried about the really interesting innovation that comes out of academic institutions, that comes out of the social impact space where that fundamental lack of clarity is what's going to stifle the ecosystem. And every single time I've woken up, entered this space and looked at what's interesting that's going on, often I'm amazed at the potential of what's coming out of the research ecosystem. And I think it would be a real shame for us to not figure out how to help support that and rather create this unnecessary regulatory overburden due to a, a lack of clarity. So really that's where I'm coming from and I accept the challenge. Right? We can't take too long. We can't make it burdensome. And I really appreciate when people show up and go, okay, then I'll, we appreciate what you're trying to do, but this is the timeline, right? In six months, I'll have the next generation of my product. Will you be ready? Hold this to account. I really do think that we can, we can meet the speed that's necessary.

[B]

Are you interested in the intersection of business and social impact? Do you want to know how corporate sustainability, ESG impact investing and more can contribute to development finance? My name is Advaah Saldinger. I'm a senior reporter at devex and I've been reporting on these issues for nearly a decade. I'm the author of Devex Invested, our free weekly newsletter dedicated to development finance. Every Tuesday we explore how companies, investors and market mechanisms are reshaping the world of development finance. Visit devex.com newsletters and join us on Tuesdays. Kristelna, I'd love to hear from you on this. Um, you know, Bilal was just outlining that a lack of clarity can be a constraint to innovation. So how, how do you think about your mandate if you do see it as your mandate in ensuring that, you know, all this regulation is critical. It's also done in such a way that innovators feel guided versus constrained.

[C]

I think the main thing is it's uncertainty that slows innovation down. It's, it's not the regulation. And I think innovators can move a lot faster when they understand what is expected, what risks matter, and how to demonstrate safety and performance of these artificial intelligence medical devices. I think it's, I always use this example. You know, if you chase 22 people on a football pitch and you say to them, go and play the game. You will have 22 people mulling around not really knowing what the objective of the game is or how to play the game and how to win. But I think once you've explained the rules of the game and how everybody can participate to actually score a goal and win the game, then people enjoy the game. Then people can play the game. People can be like, to win. For Sartra, our goal is not to police innovation, but to guide it. And we do this by creating structured, very predictable pathways where early scientific advice, risk proportionate evidence requirements and open communication is how we can actually interpret emerging standards for AI enabled products.

[B]

I love that I want to bring Miranda into this because are a couple things I'd love for you to pick up on one. Bilal was mentioning earlier that there's so much potential from what's coming out of the research ecosystem. We've also thrown around the term sandbox quite a bit, and I think people in this space get what that is. But maybe for some of our listeners who are less familiar with that term, I'd love to talk about what we mean by a sandbox. And I think on both sides, you know, ensuring what's happening in the research ecosystem actually lives up to its potential and ensuring that this work is very iterative and it sort of serves as a sandbox. I think it speaks to Wellcome's approach. So I'd love to hear from you, Miranda. What drew Wellcom to invest specifically in regulation in this space and what do you see as the links between safety and innovation?

[D]

We've increasingly recognized that if we want to change the health of everyone, it's not enough just to help create these wonderful scientific interventions. We've got to make sure they actually get to people. So I think as a funder, we are increasingly concerned with how do we help that uptake. And regulation is a key part of that, as is policy uptake, as is health system strengthening. So there are other aspects of this as well. I would say when we thought about regulation, and I want to be sort of open here, that Bilal, who was at that stage working alongside us, was one of the key people saying to us, have you thought about regulation? And in fact, we developed a project in the UK which used this model to look at how we could create better regulatory analysis. And just to be clear, we as a funder are neutral as to what the regulation does. We do not want to interfere in any way with the regulator's role. What we want to do as a funder is give them space and opportunity to both consult more deeply with the field than they might otherwise have the capacity to do and to use things like sandboxes. And I'll leave Bilal to explain the technicalities of what that actually involves and also cristel, in this particular instance, to actually try out some of the products in a safe space where the regulator and the researcher can work as sort of co thinkers and constructively critique each other. And I think this is, again, what I've really taken from the conversation so far is what we're trying to create is a new sort of relationship, which is not sort of innovators over here and regulators over here, and they're sort of trying to fight each other. Everyone wants the same things, everyone wants safe, innovative products. And so we're trying to find the best way of doing it in a very fast moving field.

[B]

So, Bilal, Miranda mentioned that you were working alongside this team prior to joining Path. You were at Wellcome and you have mentioned that you are a big believer in the importance of regulation actually to enable innovation in AI and health care. So this is sort of full circle that you're working on this project. And Miranda also mentioned that you might speak a little bit more to the importance of a sandbox approach.

[A]

Absolutely. And I imagine Kristelna has a non trivial amount of experience in this space, so I'd be curious what her reflections are as well. But I think Miranda said it well, it's not a game anymore, or it can't be if this is going to work, of just asking innovators to submit their regulatory dossiers, saying yay or nay. Right. The world is too complicated for that at present and arguably we don't have all of the right answers and we have open questions we need to answer. And so this is effectively what a sandbox is. It's an opportunity to have a conversation in an arguably safer space and one where an innovator can show up and maybe expose more than they would have otherwise with the opportunity of knowing that there are excellent researchers sat at the table. Excellently, regulators sat at the table, there are implementing partners and health services experts sat at the table. And critically, credit to Miranda and the Wellcome team for really instilling this ethos in me. People with lived experience. Right. I don't think you can come up with sensible regulation without people who have known and true experience of what these conditions are that we're trying to create products to address.

[B]

So earlier both of you were mentioning the importance of South Africa, not just being a recipient of regulatory innovation, but really an architect of solutions that others can adopt. So I want to hear a little bit more about local oversight. I mean, we're hearing this across the global health landscape right now. This conversation on African health sovereignty is really taking hold. But why specifically, specifically, when it comes to AI and mental health, is local oversight, local ownership so critical from this regulatory standpoint? And I want to hear not just only why is it so critical, but how can we ensure that what is happening as part of this new initiative can actually spread?

[C]

So, I mean, if I can jump in on that one for sopra, again, just the local oversight gives us the reassurance that the AI tools are going to be safe and they're going to be effective. For the people who will actually use that tool. And I've mentioned earlier, you know, mental health presentations, the local languages that we have, and I mean my goodness, we have 11 local languages in one country. Then also the different social and cultural norms as we also have across our different tribes and the access pathways that differ in and across the healthcare setting in South Africa. It's very diverse and really any AI model that's validated somewhere else may not necessarily translate safely here. So I think equally important is also accountability. Right? So for Safra, when a tool or a medicine is deployed all the Asian, South Africa, it's our responsibility for its performance. And that must sit with us as a regulatory authority that understands our system, our settings, our data realities and of course our populations. We cannot outsource that responsibility to other regulators who have actually not assessed tool within our context.

[B]

Lol. Anything to add there? And again, especially on how you see the lessons and regulations coming out of this sandbox as something that can spread.

[A]

I think in the regulation of generative AI for health conversation. We are so early that really the table has barely been set, the seats are empty and countries are making a play to have a seat at the top table and define what the world should look like. And I'm excited for the opportunity to step out of my bubble in the UK and the privilege of having worked with my colleagues at the MHRA who are all brilliant and say one, why not someone else? SAPRA has done some pretty incredible things during COVID thinking about some of the vaccine trials work that happened there. Why not? That's my question. And to echo Chris Elna's point around the local accountability, when the FDA asks a question about whether a product is effective and safe, they're primarily asking it about in the context of the U.S. healthcare Service, U.S. healthcare System, is it safe? Compared to an American physician, is it safe? And the reliance that many of the regulators we're talking about in sub Saharan Africa have placed on that FDA approval, that EU EMA approval, that MFDS approval from Korea, that isn't asking the question of whether it's safe in South Africa. And there's gotta be someone doing that not only, and I've said this slightly earlier for the upstream questions before it goes out into people's hands, but really downstream, because if you don't have an effective infrastructure to figure out what happens once that tool is released into that wild, if you are not finding a way to incentivize people to think about drift in performance, if you're not looking for those emergent issues. If you're not looking for effective reporting around those accidents, people get hurt. And I appreciate some may stop there. What I'm really concerned about is, yes, people get hurt, but it only takes a few people to get hurt for us to lose the trust of all of the individuals in that ecosystem who need help. And if we've lost the trust, no one's going to use any of the future technology, regardless of what we do right. And it's something that's so, so easy for us to lose and it's going to undermine the great work being done. And I think that's really why we want to step in now as early as we can to make sure that there is a space for the next set of innovations to tap into.

[B]

Miranda, I'd love to hear from you on whether you see this conversation. We're talking about the importance of, you know, local regulation for these technologies, so kind of broadening the lens beyond AI and mental health. Do you see this broader shift elsewhere of innovation and regulation increasingly being shaped in low and middle income countries rather, rather than being exported to them?

[D]

I do, and I think at Wellcome we see that shift and want to support that shift hugely. More generally. I think I talked earlier about there being a sort of transformation in mental health care. I think there's also a revolution going on in terms of how sub Saharan Africa steps up and leads the way in so many areas. And I think these two transformations are sort of coming together in quite exciting ways. So at Wellcome, we have the privilege to work alongside the African cdc, who is developing a whole new cohort of policymakers in mental health who are going to be leading the way in different models of mental health. And we have an opportunity here to create a world of mental health that's different from the world that's currently in the US or the UK or elsewhere, with all its challenges and problems with a different set of actors.

[B]

So I want to ask each of you, we've been focused quite a bit in this conversation on regulation and for good reason as we talk about the launch of Care MH and I'm excited to follow what you all do. But beyond regulation, I'd love to hear your thoughts on what other factors might be barriers to scaling AI solutions, both in mental health and more generally. So what are the other pieces of the puzzle that you're focused on or hope to see more action or investment in?

[A]

I'll jump in. I think it's payments, which is at its core a health economic issue. We need to figure out if what the interesting innovation is worth paying for, right? Not every percentage increase in diagnostic performance is something that we can afford. If we had that luxury, we'd be living in a very different world. And so as we begin to generate evidence about what works for whom, I think my real challenge to the ecosystem, and I know there are many others that are thinking about this, is what is the health economic case for us to be able to invest in this and how do we tension different products? We are only going to be able to make a a few bets as a health service that having figured out the regulatory question, now has a series of 12, 20, 50 products that we're lucky that it can pick from. We need to make sure the information is there for us to pick the right one that we can afford and is meaningful.

[D]

Just building on that, I think what I would add to that is we need to bring more investment into this space, particularly into the space of mental health. And we formed there's a coalition of scaling mental health which is led by people from the continent and then there's the coalition of mental health investment which is also a global coalition, all trying to bring in together to crowd in funding and to bring forth innovation. At the heart of it has got to be the voice of lived experience. We must make sure that the interventions that are being developed are meeting the needs of those who are going to use them and are not going to be harmful. And I just think when we talk about harms, we need to be clear that harms can happen non digitally as well. And so when we're thinking about the regulation space, let's not forget the issues around non digital harms that are happening probably all the time and in which regulators are trying to manage and sort as well. And then finally I'd say that we need to think about how we can scale these interventions, how these can be got to the maximum number of people with the minimum sort of dose and the minimum sort of harms.

[C]

I think also from my side, what I would say is that coming more from the I would say the engine route perspective is that there's actually a few few barriers that sit outside of regulation. And I think the first thing just from a South African context and where we are also as a country with national health insurance, one of these is infrastructure just having reliable connectivity, adequate computational data systems that can actually support these tools at scale for software as the regulator. One thing that we currently face with a particular challenge is as we going through digital transformation is good data, data quality. And again our AI models depend on representative, well curated data sets. So for Safra, again, you know, it's workforce readiness. So our clinicians, policymakers, regulators, we all need capacity to evaluate and use AI approach. So it's not just about the regulatory frameworks that we need to put down, but also making sure that the people are going to be using these tools. They are also ready to start using the net effectiveness.

[B]

So as we close here, we started our conversation talking about what it will take to ensure that generative AI, which has a lot of potential to help close the mental health gap globally, is used safely and responsibly. So really we're talking about ethical AI for mental health and you all are working to make that promise a reality. So if we get this right, if ethical AI for mental health really takes hold, how does that change the picture, let's say five years from now?

[D]

So my prediction is that within five years we will see mental health support in ways that none of us are envisaging now as we're sitting here. It will look completely different for some groups and that the hope is it will mean that people will have in their hands the power to choose solutions that are right for them in a safe environment where they are protected from the potential harms of those solutions.

[A]

I'll jump in there success for me in five years is there is a product that is approved by sapra, whether it is something we expect or don't, that is then available for use in another African country that doesn't yet have what we would consider a mature national regulatory authority that is relying on sapra's assessment.

[B]

Right.

[A]

If we're going to see true diffusion of this technology, bringing that locus of control closer to the continent, so that Burundi, Kenya, Rwanda can rely on a South African assessment rather than one from Singapore, the UK or the us.

[C]

And I think just from the regulator's perspective, really basically three areas where we would see this five year vision if ethical AI succeeds. And that's I think at a facial level, having earlier access to support for the regulator, having more mature frameworks, more capacity and better tools for monitoring the real world performance of these algorithms. And of course for our innovators, the landscape that is much clearer and much predictable with standards that's aligned not just regionally and on the continent, but also globally.

[B]

Great. Well, it's exciting to hear what five years, which seems like such a short amount of time, can accomplish, but it makes sense when technology is moving so rapidly. And again, really excited to follow this partnership you all are working on. That's all for today's episode. I want to thank you so much for being with us, Bilal, Kristelna and Miranda. And many thanks to PATH for their sponsorship of this episode.

[A]

Sam.