🎯 Evaluation! A make or break thing in the digital health.

I sat down with Dr Shay Soremekun from the London School of Hygiene & Tropical Medicine to talk about the very hot topic of evidence generation and evaluation.

Everyone's talking about LLM evals, technical performance, benchmarking.

But ultimately people care about impact. Yet impact is rarely a neat, linear path to a yes/no answer. Anyone who's actually implemented something in the field knows about all the other contributing factors, the daily challenges and how hard it is to move the needle on big clinical outcomes with robust clinical evaluation.

To understand why it improved care, if it did, was as important, if not more important, than understanding that it did.

Because that will help us to understand how we need to potentially modify or adapt the intervention either in the same place or in future places to be able to achieve the same success.

🌟 Who will benefit from listening to this episode?

Digital health companies : Founders, data scientists, AI/ML scientists, PMs, designers, clinical evaluation/study teams, clinicians
Funders : donors, investors
Global and public health professionals
Implementers
Researchers
Regulators

Shay explains the use of Program Theory and Logic models to visually connect your intervention, all the intermediate steps (not just technology, but people and change) to outputs, outcomes and long term impact. Making space to observe other levers in the system you didn't initially anticipate.

The work that Shay and her team have done with The Malaria Consortium and LSTM's Centre for Evaluation looking at a digital health tool for community health workers in villages and facilities showed this perfectly. Because they were intentional about observing the whole system, they discovered other factors contributing to impact and could redirect efforts accordingly.

In this age of tightening budgets, and pressure to show clean shiny KPIs, how do you make room to observe these things?

We also discussed decolonizing evaluation: capturing what's locally valuable, not just paradigms of success developed in comfortable offices and ivory towers.

So many learnings here for understanding good co-design, implementation, and how to measure what matters.

Chapters
00:00 Intro
04:15 Mozambique digital health tool for CHWs : context
14:19 Tsking a system lens approach
16:25 Measuring What Matters in Health Interventions
23:02 Resourcefulness in Low-Resource Settings
27:25 Challenges and Successes in Digital Interventions
29:34 Proactive Measurement and Adaptation
31:20 Scaling Up Successful Interventions
32:59 Revolutionizing Evaluation Practices
39:38 Incorporating Indigenous Knowledge in Evaluations
44:57 Key Recommendations for Measuring What Matters

💡 Keep learning!
If you found this episode helpful listen to:
Episode 15 : Implementation 101 and how to Fail well with Caroline Perrin
Episode 12 : Health First, Innovation second, with Smisha Aggarwal
Episode 5 : What is the right approach for regulation and evaluation of digital health technologies?

I have also written a Substack article summarising what I learned from this episode. Check it out for easy reference to the concepts outlined.

Found it useful? Know someone who is struggling with this very thing? Share and keep raising the profile of people impacting underserved community globally.

References and links to things we spoke about

inSCALE papers:

https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000235

https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000217

Link for upSCALE (roll-out of project):

https://www.malariaconsortium.org/projects/upscale

https://www.malariaconsortium.org/resources/integrating-upscale-into-the-ministry-of-health

Study protocol for Uganda and Mozambique:
https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-015-0657-6

Links to Chilisa Bagele – one of the most foremost voices on decolonial evaluation methodology so a good start for anyone interested (talk at LSHTM):

https://www.lshtm.ac.uk/newsevents/events/decoloniality-and-indigenous-methods-global-health-evaluation-professor-bagele

Centre for Evaluation Website

https://www.lshtm.ac.uk/research/centres/centre-evaluation

About Dr Shay Soremekun
Dr Shay Soremekun is an epidemiologist and co-deputy Director of the Centre for Evaluation at the London School of Hygiene and Tropical Medicine.

Her research in child and adolescent health and development covers trials of low-cost disease prevention programmes, and identification of risk factors and mitigation strategies for poor developmental and economic outcomes in this group. She is a member of the UK Government Evaluation and Trial Advice Panel (ETAP), and sits on the steering committee for the John Snow Society. She lectures at postgraduate level on topics of evaluation, epidemiology and public health, and has developed and organises an MSc module in Study Design.

LSHTM page: https://www.lshtm.ac.uk/aboutus/people/soremekun.shay

Evaluation level up : Measuring what matters