A (5:30)

And it sounds like this has to do with risk. Like the higher the risk, the more you want to put these into place.

B (5:35)

Exactly. We call it calculated risk. So as a product manager you need to know what is non negotiable and what are the risks that can't be jeopardized and just work around that.

A (5:47)

So Mariam walked us through how AI agents work, why hallucinations happen, and why guardrails and human oversight become critical as risk increases. That idea of calculated risk is especially Important in regulated environments where mistakes can have serious consequences. Next we'll hear from Meg de Armbruster, head of product at Natural Cycles, who shares how her team builds healthcare products where regulation, privacy and compliance are embedded directly into product development. Here's Magda.

C (6:17)

We have quality assurance members within our team that work alongside of product development and they do flag to us early if we try to do something too risky or too out there. Our quality assurance is like inbuilt into product development processes. Whenever we brainstorm like a new idea they are part of that brainstorming. They are part of our our team meetings where we present like new designs or new features and they give us feedback so we take that into account right from the beginning. We don't work in silos that first the product team works and then we hand it over for sign off and then it comes back after a month of limbo. We work alongside the regulatory and the compliance team with like frequent meeting and things and also just talking to each other.

A (7:16)

There is a very big topic in the United States right now about that as well and I've seen Natural Cycles and apps like it in the news, especially when it comes to the government who our choices have been more and more restricted here. A lot of news articles talk about how they are going to start the government's going to start looking at these apps, taking people's data to find out if they had an abortion or if they prevented pregnancies which could lead to legal action. How do you think about first of all is any of that true and how do you think about data privacy when it comes to such a sensitive topic like this, especially in a political climate like we have today.

C (7:54)

When it comes to data privacy and data security, we are a paid app, which one reason for us being a paid app is that we do not sell our users data. Our users data is in their hands is under their control. They are in charge of it. They can delete it, they cannot add any data. They don't need to do anything. They're absolutely 100% in control. If they don't want us to use their data for research, they don't have to. It's absolutely all in their hands. All of the practices around data privacy and data security they are under a NC Secure flagship product in our company and one of those is also even an advanced identity protection called Go anonymous mode. Should women want to go anonymous, they can. And this means that not even us cannot identify who they are. So in an unlikely hopefully event of a subpoena from the government, we will not be able to give the users data because we don't know who the data belongs to. People often think that if you're a regulated medical device and if you need to follow like a very strict process, it can slow you down. But I think for us it's actually the opposite, which I think it's pretty cool. We have, we have a very well defined quality management system that has been with us since the beginning. So it's like a foundation of what we do. Whenever we like tackle a new feature, we can look at the product requirements at the risk. So we do have a baseline to start with, which is an excellent thing to have. Also when it comes to being a regulated medical device, we do need to document quite a bit which also might seem like a hurdle. But if you have it incorporated in your process, it's actually very seamless and it does help to document your decisions. User insights, feedback that we get so that you can refer to it later on. And we do innovate actually quite a bit. One aspect that I also wanted to touch upon is that we do have a great culture in the company where everyone feels like they really can contribute to the success of the company. And if someone feels like they can impact the mission or the product or the vision, the they will be more likely to innovate and to come up with great ideas. And we have an amazing team that, that does that basically on a daily basis. So when we do have a new product and we innovate on it, we do bring also our compliance and our regulatory teams early on and they also shape and they also can help us shape that product. So it's really working together with the regulatory team. It is not a blocker. It actually it just provides us with a framework that enables the innovation.

A (11:00)

Magda showed how bringing regulatory and compliance teams into the process early can actually create clarity and enable innovation rather than slow teams down. But even with strong processes in place, AI introduces another set of questions around cost scale and long term sustainability. For our final perspective, we'll hear from Jessica Hall, Chief product officer@justeattakeaway.com she talks about how product leaders should evaluate those trade offs from unit economics and simplicity to governance and building lasting capability. So this is Jessica on what it takes to make those decisions responsibly.

D (11:37)

I think there's always a whole load of inputs and it really depends on the situation. So there's a cost piece in this. We are excited about AI and there's lots of conversation about it in the industry. But implementing LLMs. Training LLMs and running them can be really expensive. And if we're talking about something that has a very small impact, it might be great for your customers, they might really love it, but it might not commercially really move the needle. Then you have to ask the question, like, is this the right investment to make or are there cheaper alternatives that are as good or 90% as good? And that's always a judgment call that you have to make. I think the other piece is just keeping it simple. And that's one of our, one of the things that I hold really true and I challenge my team on as well. Sometimes with the AI solutions, we can be guilty of over complicating things or over engineering it. And actually simplicity is the answer. So asking yourself, like, is this the simple answer? Is this the right thing to do? That's really important. And then I think once you get to that point, it's then a question of what's your tech stack? What data do you have? Can you, do you have enough data that you can make this meaningful and you know that you can run it effectively? You have to ask yourself all of those kind of things. How accurate is it going to be based on the information that you've got? And is that a risk you're willing to take? Because with generative AI, there is also a risk element of this because we are still learning and the LLMs are learning and it's changing all the time. We've seen some mishaps in the industry and there will be more to come, I'm sure. But it's all of these factors together and I think I will go back to what I said also about hype. Not getting dazzled by the exciting potential, but actually asking like, what is the problem we're solving and does this really solve that problem?

A (13:35)

I've been playing with LLMs on, on our side too for like our product institute training to see if we can make things more interactive. And what surprised me about it was some of the stuff that you just said too, that I don't think a lot of people are looking at. And one was the cost side of it, right? There's a scale and I think a lot of people are like, oh, LLMs, it's gonna, it's gonna reduce the cost. You're like, for some, if you want to self host, for example, an LLM on aws, like that's expensive. Sure, it could replace a couple thousand people if you have that, but if you don't and you're doing it for the wrong reasons, like that's expensive. And I don't think people are actually looking at it.

D (14:10)

Yeah, the cost side is quite interesting and depending on the size of your business and what you're going after, it's definitely worth looking at. That is one of the things we assessed when we were doing our AI assistant for customers. What is the cost per chat? And when you're thinking about conversion, customer acquisition, which I'm sure many people who are listening will be very familiar with that, then you have to think about what is the cost of a chat, what's it going to do for your first order, second order and your conversion metrics. Are you seeing it as a way of doing acquisition? And have you built that cost in? And it can be surprisingly expensive. And I think that probably the other thing is it isn't just the cost of the technology, it's the cost of maintaining the team that needs to continue to work on that, the skill sets that you need to build. And that's something I also take very seriously, not just with AI, machine learning and other technology in that kind of space, but are you building a department that is fit for the future? Are you training your colleagues, your team on the things that they need to know to be effective into the future? And as you adopt these technologies, it isn't just about adopting the technology, it's about creating the entire ecosystem of your department that's able to take it forward.

A (15:24)

Yeah, it's building in a capability to your business.

D (15:27)

Yeah, it is. And it's not a reason not to do it. But that expense piece, there's lots of other things around the data governance and how you govern it. And bias is a great topic to talk about on this as well. And how do you tackle all of those parts of the solution as well? I'm not for a moment saying don't do it, just you've got to go into it eyes open and not be dazzled, but really make the right decision for you.

A (15:53)

You just mentioned something too about bias, and we have talked a lot about bias in AI. And it's a computer, it's generating stuff based off the data you give it. How do you help run the governance around the AI that you have and how do you mitigate for that risk? You also talked about it giving like the wrong answer or doing something that's not desired because they do hallucinate. What do you do to manage risk around there?

D (16:14)

Yeah. So all of these things, I would say we do have a robust data governance kind of policy and procedures already in place. And AI falls into that category already. So we already have much of that in place, but we have developed that out further to make sure we really get this right, or at least that we are giving it the best chance we can to be on top of some of that stuff because it is new and is developing all the time. So we do have a cross functional team of people who look at these solutions from all perspectives that we have data protection, office, legal, tech, product, various different people who come together regularly to talk about how are we using it, what are we doing, what are we learning, what's the industry saying and making sure we're really on top of all of that. And we try to be transparent with customers as well. So that AI assistant that I'm talking about, we give the customers the choice to opt in to us analyzing their chats. We want to, because we want to know what they want, but we're giving them the choice and we're being transparent about that. In the first chat that they have, we ask them that question. So I think that's also really important. Not only that, we've set up an AI guild so anyone with an interest across the entire organization, not just product, not just tech people, but anyone, can get involved in the AI guild. And through that guild we are educating people on the foundations, the policies, safe use or correct use within our guidelines so that we can make sure that we don't allow this to go unchecked and that we are really eyes open to the risks and the way we use it. When it comes to bias and inaccuracies, it's difficult. We've created a cross functional team that work on the AI system that's made up of a number of different people, different backgrounds and is quite a diverse team. And I think it is very important to have diverse teams building solutions not just for AI, but especially for AI. Because if we don't think about how we train the models, we risk exacerbating some of the bias that we see in society today. I am not for one moment saying that we have solved that problem or that we are the perfect business and we know exactly how we're doing it, but we are giving it a very good go. And we are really aware that's something we have to be on top of in terms of inaccuracies. It's difficult. The longer the conversation goes on, the more likely the AI is to hallucinate because of short term memory and all of the things that, you know, how LLMs operate. From my perspective it's, it's not just for the inaccuracies. It's also better for the customer if we answer their question quickly and they don't have to ask five times for the meal that they want, but they can get a pretty good answer straight away. And speed is important too in that actually initially we found that our assistant was too slow, customers were waiting too long, so they were asking again, and that's much more likely to lead to inaccuracies. So those are some of the things we've done. But I think just like everybody else, we're on that journey and we're learning and developing alongside the technology and that's what makes it really exciting as well.

A (19:31)

Thank you so much for listening to the Product Thinking podcast. We'll be back with another episode soon, continuing to share practical insights on how product leaders think and work. If you want to hear the full conversations with Miriam, Magda and Jessica, check out episodes 241, 251 and 199. Make sure you like and subscribe so you never miss an episode. We'll see you next time.