← Scrum Master Toolbox Podcast: Agile storytelling from the trenches
Scrum Master Toolbox Podcast: Agile storytelling from the trenches
When AI Decisions Go Wrong at Scale—And How to Prevent It With Ran Aroussi
00:41:05
Loading summary
Transcript51 lines
- [00:04]
A
Hey there, agile adventurer, just a quick question.
- [00:07]
B
What if for the price of a.
- [00:09]
A
Fancy coffee or half a pizza, you could unlock over 700 hours of the best agile content on the planet? That's audio, video, E courses, books, presentations, all that you can think of. But you can also join live calls with world class practitioners and hang out in a flame war free and AI slop clean slack with the sharpest minds in the game. Oh, and yes, you get direct access to me, Vasko, your Scrum Master Toolbox podcast. No, this is not a drill. It's this Scrum Master Toolbox membership. And it's your unfair advantage in the agile world. So if you want to know more, go check out scrummastertoolbox.org membership, that's scrummastertoolbox.org Membership. And check out all the goodies we have for you. Do it now. But if you're not doing it now, let's listen to the podcast.
- [01:12]
B
Hello everybody. Welcome to a very special bonus episode on AI transparency and alignment. With of course, our own interests as humans. We've spent the last few years asking what AI can do. But the next frontier isn't more capability. It's something perhaps not so interesting for most people. Less glamorous and far more dangerous if we get it wrong. The question is, what happens when AI systems make decisions no one can explain at scale? And today I'm joined by someone who's deep in the machinery of production, AI building, breaking and rebuilding for the real world. We're going to talk about why observability, transparency and governance may be the difference between AI that empowers humans and AI that quietly drifts out of alignment. And we'll explore what all of that means with our guest, Ran Arousi. Hey Ran, welcome to the show.
- [02:09]
C
Hi, welcome. Thank you for having me.
- [02:13]
B
Yeah, it's a pleasure to have you. So let me tell you a little bit about Ran. He's the founder of muxi, an open framework for production ready AI agents. He's also co creator of very cool package that I use quite heavily called why Finance? Check it out. And he's the also the author of a book, production Grade Agentic AI From Brittle Workflows to Deployable Autonomous Systems. And I haven't read the book yet, but I'm pretty sure the autonomous part is very important.
- [02:46]
C
Yes, the autonomous part is, is very important. And that's one of the main reasons why I decided to write this book. It all started with, with a rant of mine about workflows in AI, which is Kind of an oxymoron and kind of led me through this path of writing a 650 pages book about it.
- [03:08]
B
Oh wow. Well, we'll learn all about the things, at least the most interesting things that Ran discovered and we'll talk about alignment and governance. But before we dive into that, I want to start with you, Ran. You've built tools that millions of people rely on. I just heard from you that Y Finance is now at 10 million downloads a month, which is mind boggling. What's your origin story? Pardon me? With this AI and systems thinking.
- [03:37]
C
Yeah, so obviously the origin story was I'm quite old, so I've been around the block for a while. I started when I was 13, which puts me at around 35, 36 years ago. And I've been through from Basic as a first programming language, all the. Through the EdTech space, I just jumped 20 years. But all through the EdTech space and writing algorithms for online media exchanges that led me to. To writing tools for online trading because I found a lot of similarities and I was kind of tired from Google and Facebook taking over the advertising space online and I was and still am doing a lot of independent trading and algorithmic trading. I'm all about automations and that's why I've created Y Finance, which is the library to download the stock information. And I have other libraries in Python for statistics and analysis of performance of portfolios. And obviously when the new shiny toy came around, which is AI a few years back, as a bonafide geek, I jumped heavily in and I've played around with it, lots of coding and vibe coding, although I'm not really doing vibing, it's more of an AI assisted coding.
- [05:22]
B
I like you use that term because actually we have started the whole series of episodes here on the podcast. Exactly. About AI assisted coding. Which should get you back to get your take on that because there's a. There's a lot of meat in that.
- [05:36]
C
There's a lot of meat in that. Exactly. But yeah, I essentially graduated from building demos in AI and I also run a company where we do software development for clients. And there, as you can imagine over the past two, three years there was a lot of emphasis on AI applications and AI powered application. So I brought a lot of what I know from systems engineering and deploying products at scale to the AI space and that kind of led me to moxie which is going to be out any day now, literally polishing the documentation website and. Yeah, but that's essentially what I'm mostly focused on Right now is bridging the gap between amazing demos that you see everywhere and one shot applications that you see people kind of showing around on X to making it real deployable system that you can trust to survive real users experience and more importantly understand what's going on under the hood.
- [06:49]
B
So what was the moment when you realized that developing or building products with AI was a completely different challenge than deploying them safely?
- [07:01]
C
Yeah, so the main uniqueness in AI is its non deterministic nature and just, just by design it doesn't follow a programmatic workflow where you build your if then loops and condition and kind of have your own journey of how the data should flow. With AI, you've noticed as I mentioned, that they're really cool demos where you just import something like Agno or Crewai or some sort of a library like that and immediately you have a chatbot and some reading some files and you have a support agent and it's really cool and. But now taking it into production is quite a journey. And when I say to production, I come from an industry, spent a lot of my adult programming life in the edtech space where we had to deal with lots of traffic. And when I say lots of traffic, I mean serving 3,4 billion ads every day and handling over a million concurrent users and hits to the server. So. So when I'm thinking about production at scale, these are the numbers that I'm kind of looking at. And I noticed that I watched well designed agents makes perfectly reasonable decisions based on its training, but in a context where the decision when the decision was catastrophically wrong and there was really no way of knowing what had happened until the damage was already there. So building it is not, as you mentioned, it's not really about capability anymore. The agents are quite capable, AI is quite capable, obviously it's going to get even more capable. But I think that deploying it safely right now is more about deploying it reliably and about visibility and audibility. Especially if you are an enterprise and you want to embed AI into your infra and your internal processes, it has to be audible, you can't just wing it.
- [09:32]
B
And that's one of the things you talk about, right? Like you already mentioned that it's not about capability, bigger models, better benchmarks, etc. It's about observability. So when you talk about that, and especially when you talk to clients, what do most leaders misunderstand when they perhaps focus only on what AI can do for them instead of how it behaves in production and all of those other aspects related to the practical deployment and safe deployment of these tools.
- [10:04]
C
Yeah, a lot of people come with their own biases. Not, not in a bad way. But when you chat with Claude or ChatGPT or whatever it is, you chat about your own life and your own stuff. So it's predictable. And there, I assume there's a limit to the, to your imagination of how many different examples you can test where you're coming up with scenarios for AI. And when you chat with ChatGPT and you ask the same question and you think you're being clever by asking it in different ways and try to trick it, you're essentially going to get very similar results. So you think that you are under the assumption that AI is somewhat predictable and somewhat observable. But what happens when you deploy a system in production, you have users with unimaginary different use cases and different problems and different way of phrasing themselves. And maybe they're not native English speakers, so their English is constructed in, in a Hindi way or in a Flemish way that, you know, the words are not in the same order as they should be. And it's a very common mistake. And then the AI suddenly start bringing in completely different answers and behaves a lot differently. So while in traditional software bugs are reproducible, if I come to you and say, okay, when you click on this button and you go there, you're going to see a glitch. And when you do that, you're going to see a glitch because it's software, it's very deterministic. But if you run the same code with the same input with an AI system, you are going to get different results, even if it's slightly varied results, even from, if from a human perspective, it looks the same and it sounds the same, but because your software expected a specific output format, then it's not working anymore. So that was the main kind of the moment that I've realized that that was the gap that clients really had an issue with. Because from, as I mentioned, when they chat with ChatGPT, where they go there, it pretty much works all the time for them.
- [12:39]
B
One of the things that is like your answer is already pointing to is this idea that first of all, the input space for the AI systems is practically infinite because it's language, you can never test it. Right? Like there's no way we can test it.
- [12:56]
C
Right?
- [12:57]
B
But the easy answer to that, and on the security side, which is what I'm more familiar with, we already have an answer for that by the way it doesn't work very well, which is guardrails. Yeah, so what are you talking about? So you talk about observability and what else? Because it's not enough to know what happened, we also need to prevent catastrophic consequences.
- [13:18]
C
Yes, so you, yeah, so you mentioned guardrails, which is it has to be a multi layered system that protects output. So first off, you need to, you know, when you develop a software and your software worked with Python 3.10 patch 7, then you're not gonna just magically do an Upgrade to Python 3.13 before testing it thoroughly. So even here you want to be able to lock the version of the model that you're working with, because that's the model that you've tested. So if you've tested with the GPT5 mini, don't jump to GPT5 before fully testing it because the result can completely vary. Not to mention if you switch to Grok or Claude or whatever it is, you're going to get completely different decision makings. So that's, the first layer is make sure that when you do the testing, use AI to generate a bajillion different options for the testing and make sure that you're always testing on the same model and you're locking it. You know that, okay, this is what I can expect. And then you can also. And then the second layer is that you also need to put in the guardrails. Never let, for example, personal identifiable information come back to the user. Don't share this type of information, don't answer question about how do I kill myself or stuff like that. You want to be able to put the guardrails in place. Usually it works a little bit better if you threatened it. It's surprisingly. But, but, but it works. I, I'm not sure it's going to work forever. It works now, but then you also have to put in deterministic filters between the AI and what you, you, you get back to the user.
- [15:30]
B
Like language firewalls, basically.
- [15:32]
C
Exactly, exactly. So the simplest example is like those profanity filters that we used to have on, on forms. So something like that make many patterns as possible to block personal, identify information from coming back to the user. Even if the AI gave you a detailed plan on how to kill yourself, make sure that using deterministic methods, you just block that answer and you come back. Or maybe don't even use detection to not even send this answer to the AI in the first place. So that's kind of the third layer is the deterministic system that you put in front and behind the user request to and from the LLM. And finally the most important thing is that you have detailed logs on what had happened. And when I'm saying logs right now we've used a very simple example of a conversation. But with agentic AI you have decision making, you have task decomposition, you have tools that it decided to call and what data to pass to them and what to do with the data that came back from those tools. So there's, and not to mention if you're using, if your agents are using A2A and communicate with completely external agents as well, that's a lot of things that you at least should be able to trace back. You should be able to have to be able to build a knowledge graph and let's call it a workflow, a timeline of the user's request so you can then on mass feed it back to some sort of a system again, a hybrid of deterministic and non deterministic to extract analytics and information of where you detect patterns. Okay. I've noticed that whenever users ask something with the word I don't know banana, it decides to use the GitHub tool for some reason. Let's figure out why. Oh, it's probably thinking about nano banana and is looking for repositories. But you have to collect this data. That's why you don't just build the system and go to production. You have to have thorough testing. That last part of trillions of, of parameters we're going to have in AI models quite soon it's going to be in the trillions. So yeah, you never know what you're going to get.
- [18:17]
B
So that last part almost like reproduction and analytics system, right? Like so that we can reproduce interactions and all kinds of interactions, internal and external and then an analytics system that helps us to find patterns and eventually detect, I guess we could call it workflow blind spots. Right? Like things that happen that nobody expected. And because the input space is infinite, the interactions of those inputs are of course then going to be infinite to the power of X, where X is very large, very, very large.
- [18:53]
C
So it's almost true infinite. Especially if you're dealing with users and they can provide their own token to access the, whatever the zero MCP tool or the GitHub MCP server. So it's infinite to the power of X, to the power of how many users you have. It's getting quite ridiculous quite fast.
- [19:18]
B
Yeah. And therefore, I mean for me, what's kind of the insight in this conversation is that observability now is a completely different challenge because, you know, even in a very large deterministic system, the, even with a very large input space, the output space is going to be very limited because the system is deterministic. So the number of outputs is limited by definition. But, but now we're talking about observing potentially infinite interactions, which is impossible. So what have you found that really helps people that are right now trying to deploy this system safely and facing this problem of understanding that actually we can't actually control this once it's live? What we can do is learn from what is happening and iteratively improve it over time. Right. So what are you doing and what are others doing that helps us to try to bring some sense of not being in control, but rather not being exposed to catastrophic problems?
- [20:24]
C
Yeah. So sadly I wasn't able to find something that truly works with agentic workflows. And there are a lot of tools that are doing that to some extent, but they're, let's say that they're more, they're better at LLM, at CHAT and at conversational observability rather than agentic observability. And agenda observability is like everything that I've explained. But you also need to have some layers to it. So engineers need to have access to full level of data for full debugging and these need to be Trustworthy individuals with NDAs and full checks on background checks, et cetera. And then you have to have more of a managerial where there's no personal information, you just show flow based on topic. It's kind of a tagging system to make sure that you follow that. But we've mentioned that unlike a simple conversation where it's more about evals. Okay, I want to see that. I'm basically getting the right answers for the request here. Your request. If you say something, what's. I know, what do I have to do today? Okay, now let's take the start with the user synopsis. First of all, who, who is you? What when you say what do I have to do today? Who are you? And based on that I might decide that I'm just going to read you a bunch of Slack messages that you had and kind of important emails that you have to respond to answer to. But may maybe you work in the marketing department. You need to grab a full data of everything happened from the, the telemetry of your software or whatever it may may be. Or maybe you're a managerial. So we need to go into the accounting and finance and check more, more to so it, it changed based on not just what you ask but on who you are in the organization and what tools have we connected and gave the agentic workflow. So the first thing is when you say that happens in the system, when you say what I have to do today is we need to understand who you are. We need to based on that context check which tools are relevant for us to use. Go fetch all the data from those tools. Learn about your person, your preferences. Like when you ask for the weather, do you want the degrees or do you want a full story about the participation and what's going to happen throughout the day and whether or not you should take an umbrella or just want oh it's 10 degrees. So how the style of the communication that you want what would have the system learned that you don't really care about? Right. So for example, you know, want to know everything that happens in finance but for people who are looked you up on Slack, all you need is three people are waiting for your answer. Just go to the catch up tab on and slack on and figure it out. You don't want the details. So there's a lot, a lot of information here that can significantly change the decision of what this, of what the system is actually going to do. It's a dynamic workflow that just got built based on who is talking to the system and their preferences and their history and the tools and everything. So it's a completely different workflow. If you ask what do I have to do today? And I ask what do I have to do today? Yeah so that's so you have to be able to and that's sort of kind of an eval type of observability. Let's say what workflow was even built based on that question. Make sure that those workflow makes sense and kind of are in compliance with what we expect. And then we have a kind of a complete set of a can worm that we need to trace which is if it was actually executed as expected. Not just planned as expected, but executed as expected.
- [25:03]
B
So observability brings all of these kinds of questions and I'm sure you talk about that in your book. So make sure everybody to check out production grade agentic AI from brittle workflows to deployable autonomous systems.
- [25:19]
C
The link is in the productionai book.com that's simpler, easier to remember.
- [25:24]
B
There you go. Productionai book.com but the other aspect beyond observability is governance and you know engineers tend to get nervous. I work in the security field so for sure we all get nervous when we talk about governance because it is critical. And in our case when we're talking about AI, of course it's keeping AI and the human aligned at scale. And there's a phrase that you use that I really like. Governance isn't about control, it's about keeping people in the loop. So when you think about this system, let's just look at this thought experiment. You started with the workflow that starts with what do I have to do today? Where should humans stay in the loop even as AI systems get more and more autonomous? Like the Slack example is simple, but what if the person sitting at the desk asking the question is the CFO who has a bunch of 1 million plus transfers to approve because of either purchases or investments or whatever.
- [26:28]
C
Yeah. So the governance need to and you said it quite well that it's about keeping human in the loop. It's not about control. You don't want to be able to take over the human and make them completely oblivious as to what is happening. And you don't want the the human to be able to have to approve every single thing that the AI is doing. So there are several layers. So like when you design a product, some things are quite self explanatory. Hey, we need a login page that does a login with Google. Fine, everybody knows what I mean. But sometimes there are follow up questions. Okay, is there a specific stack that you want to use? A specific provider that you want to use? So here as well we can have follow up questions. So you mentioned the CFO. You have 10 transfer in the sum of a billion dollars to approve today. Do you want more information? Do you want to start kind of going them one by one and approve them one at a time? And that's great. So the human can say no, I'll get to it when I'm at my desk. Or the human can say sure, let's start approving them and then the system can start reading them. But here we need to have a deterministic. I still wouldn't be comfortable to approve a transfer to AI and have it send the money. Okay. At most I will allow it to you. You mentioned that you're doing trading it. Just put in the order, let me go and I will check that the that all the transfers are correct and I'll just have to say yes, yes, yes, you got all the details right, everything's fine. You have a link back to the why this company is going to get this amount of money today. So I just need to approve it because you don't want to approve and depending on your organization and your budget comfort, you don't want to approve $100 or $100 million transfer and now discover that it was sent to the wrong account. And now you have to chase the money all around the world to hopefully get it back. And even if the transfer gets declined and comes back to you, that could take a good few days. And now your delay of sending the original payment to where it needed to go. So that's where, for example, a human in the loop type of thing should go to. An AI will go to some extent to automate things for you, but then the human on some things will have to approve it. I don't need to approve an automated email that answers support tickets. At worst I can make the AI cc me if it wasn't sure and I can always jump in and say I'm sorry, that was an automated and kind of course correct, but it's different based on the use case.
- [29:58]
B
So what this suggests to me is that we also need to start thinking about an AI as an interactive partner in some of our own human processes like approvals and decision making and so on. So that we get the benefit of what the AI systems can bring us. But then we get the governance. I think governance better than control is definitely a better word. So that we get the governance that we would expect even in a fully human system.
- [30:26]
C
Right.
- [30:26]
B
Where there's no AI involved. So is that how you see it?
- [30:31]
C
That's definitely how I see it. Because if we're looking at the long term vision, what everyone wants is a Jarvis. Right. And there are some things that Jarvis just do without asking for permission either reactively based on a request and then it just figure out what it needs to do and it doesn't ask for permission for every step proactively. So for example, we have an AI that would detect that I'm home and you know, if we used to have an Alexa turn on the lights now the AI can do other stuff for me when, when, when I get in the home. But there are other things and that's where guardrails come in and deterministic stop stop points where I have to be involved like that. So I wonder even if with coding agent before deploy to production, you want to have a human in the loop, even if you trust the agent completely to do all the writing, at least check the code before it goes to production, if not the code, check the result, you know.
- [31:42]
B
Yeah, I think with coding is even a little bit more specific. I was just thinking as you were describing the Jarvis example, that we're getting to the point where we need to start learning and trusting how the AI system works. Right. So the frequency of interaction and the success rate of those interactions are going to move us closer to using the system or not, depending on how well that goes with coding. I'm not so sure because as you said, coding is one of those points where a small change can have a catastrophic impact. So you want to have, even for humans, right? Like you have automated systems integration beta deployments or staging deployments and then production deployment. So even for us we have all of those. Exactly. Even for us we have all of those stage gates. But for internal processes in organizations, I'm thinking that AI may become over time just like any other individual. Right. Like you need to learn to talk to them, you need to learn to trust them. And that's not going to be done in one interaction. It's going to be like I guess a sequence, a set of multiple and hopefully mostly successful interactions.
- [33:01]
C
Yeah, it's very similar to, I mean our aspirations have a human like counterpart and the AI. But even humans, you don't just meet someone and you trust them completely. You have to work with them, you have to make sure that their performance is on par and everyone knows that and have a good friend that you wouldn't trust for anything other than fun stuff. And you have those co workers that you know that you can say, you know what, you're in charge of a company, I'm off for vacation. And you know that everything would be fine, but it took time. You probably don't know this person for a month, you probably know this person for years and you've worked with them and the first time you went on vacation they had like 10 calls with you and then slowly got around to, I'm only going to call you if it's really urgent and I really don't know what to do. And you build that trust where you can rest comfortably knowing that things are being taken care of. And you cannot have that trust with something that you don't know what's going on with. So even if you don't know how AI works and how system works, you need to at the very minimum build that trust in the performance and the result. And that wouldn't happen without starting with a large amount of human in the loop interaction. And slowly as you build trust, you know, you know we've done this a thousand times, you don't have to ask before permission before you do X. But I still want to be to be able to approve every. Yeah, exactly.
- [34:46]
B
All right, we're getting close to the end Ran. So the book that Ran wrote and you should all check it out is production grade agentic AI. The link will be in the show notes so make sure you, you check it out. But Rand, for people who want to dive deeper into the topic, what's one resource you would recommend? Could be a book, a paper framework, a person, a YouTube channel, whatever that is.
- [35:12]
C
Honestly I would and I would recommend a bunch of a bunch of books in a minute but I also want to kind of give a meta recommendation of which is less about AI and more about systems engineering, distributed system and operational excellence. It could be non computer related at all. It's about how to run large scale organization and stuff like that. So that's really going to be my meta recommendation. One of the really the books in that regard that I would highly recommend is Thinking in Systems. It's really, it's by something Meadows. Yeah, I forgot dawn or Meadows.
- [36:04]
B
Donnella.
- [36:06]
C
Yeah right, that's right. So I would definitely recommend this book. It teaches about feedback loops, leverage points, it's really essential for everything want to understand why agent system behave unexpectedly at its scale while still not being about agents which is really a really cool way of looking at it. From particular implementation Designing Machine Learning Systems by Chiphun and sorry I forgot the name of the author. I'll send it to you after the call. But writing LLMs from scratch really important. Yes, you're gonna have. You're gonna program a really bad LLM but you're gonna understand how it works. So you can probably get to a good level of GPT 2.5, maybe 3o level but you're gonna at least be able to fully understand how LLM works, how they're making decisions and once you understand the statistics probability of it, you'll be able to trust it less on some things and trust it more on others. So that something that I would definitely recommend as well.
- [37:36]
B
And we'll put all the links to those in the show notes.
- [37:38]
C
Yes.
- [37:38]
B
And how about you Rand? Where can people go to find out more about you and the work that you're doing?
- [37:44]
C
I'm on X a lot so you can find me on X at Arusi A R O U double S I and yeah then if I have something important then I will link from there. Currently I'm working on Moxie which is a project that hopefully will get people as excited about it as I have been working on it for about a year now. And yeah it's one of those mega projects and hopefully it will bring what Docker brought to deployment into the AI world. It's a very cool thing.
- [38:24]
B
We'll link to that also in the show Notes make sure everybody has access to that. Rand, it's been a pleasure. Thank you very much for your generosity with your time and your knowledge.
- [38:33]
C
Thank you very much for having me. I'm really enjoying talking about such things and with like minded people who understand the importance of everything AI.
- [38:49]
A
Alright, I hope you liked this episode, but before you hit next episode, here's the deal. This podcast is powered by people like you, the members who wanted more than just inspiration. They wanted real tools and real connection to people who are practical Agile. Every day we're talking access to over 700 hours of agile Gold, CTO Level Strategy talks, Summit keynotes, live workshops, E courses, Deep Dive Interviews, books, and if you're into no Estimates, we got the pioneers of no Estimates in those Deep Dive interviews as well. Agile Business Intelligence, Creating Product visions, coaching your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners, plus a private Slack community which is free of.
- [39:49]
B
All of that AI slop you see everywhere.
- [39:51]
A
And of course without the flame wars. It's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together.
- [40:05]
B
So if this podcast has helped you.
- [40:07]
A
Before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have many so much for you. So check out all the details@scrummastertoolbox.org membership because listening is great. It's important. But doing it together, that's next level. I'll see you in the community.
- [40:39]
B
Slack we really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes? Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.