← Scrum Master Toolbox Podcast: Agile storytelling from the trenches
Scrum Master Toolbox Podcast: Agile storytelling from the trenches
BONUS Why More Code Doesn't Mean Better Software — And Where AI Actually Helps Your SDLC With Mooly Beeri
00:40:00
Loading summary
Transcript58 lines
- [00:04]
Vasco
Hey there, agile adventurer, just a quick question.
- [00:07]
What if, for the price of a
- [00:09]
fancy coffee or half a pizza, you could unlock over 700 hours of the best agile content on the planet? That's audio, video, E courses, books, presentations, all that you can think of. But you can also join live calls with world class practitioners and hang out in a flame warfree and AI slop clean slack with the sharpest minds in the game. Oh, and yes, you get direct access to me, Vasko, your Scrum Master Toolbox podcast. No, this is not a drill. It's the Scrum Master Toolbox membership. And it's your unfair advantage in the agile world. So if you want to know more, go check out scrummastertoolbox.org membership. That's scrummastertoolbox.org Membership. And check out all the goodies we have for you. Do it now. But if you're not doing it now, let's listen to the podcast.
- [01:12]
Hello everybody. Welcome to one more bonus episode of the Scrum Master Toolbox Podcast. And on this episode we have joining us a previous guest. So I want to say welcome back to the show Mouly.
- [01:27]
Muli
Good to be here again, Vasco.
- [01:30]
Vasco
So check out the previous episodes with Muli with the link in the show notes so that you can also find out more about the work that he's doing other than the topic that we have today. Because Muli wrote the answer to where should AI fit in your software development lifecycle process? And the answer is deceptively simple. Find where you're already underperforming, then measure before you automate. This part is important and in this episode we break down why most AI adoption software teams fail and what is a disciplined approach that you could take and what that looks like. But of course we need to get a little bit of context of our guest today. So Muli, you've been helping software organizations improve for over 25 years, long before AI was even a topic like it is today. I mean, AI existed already. It was just very different from what we have today. What got you into this work of improving software organizations in the first place?
- [02:37]
Muli
Yeah, great question. Well, AI existed in the science fiction books that I read as a child.
- [02:43]
Vasco
I remember Asimov with what is the three laws of robotics? Right?
- [02:49]
Muli
Yeah, yeah, but there was actually artificial intelligence in some of the early Lajmi books and others. But anyway, so I've been an engineer for as long as I can remember and I always liked solving problems, complex problems, and always had a passion for finding out process areas that need improvement. And understanding patterns. Probably I'm a very artificial version of artificial intelligence because I've been identifying patterns way before AI was doing that. So when I started out, one of my first jobs was at Microsoft and I got the opportunity to work in engineering excellence and I just loved it. So I was happily debugging process areas and finding areas for improvement, debugging processes,
- [03:47]
Vasco
debugging people's communication skills, I bet.
- [03:51]
Muli
Well, it's all over the place. But this was really fun because it was such large scale in every single simple step. A simple improvement would make the lives of so many people better and the code better and the products better. And later on, I think I was just lucky. I got the opportunity to build the first software center of excellence for Philips, where I just built upon my passion from Microsoft, and it was working well. I spawned it on to a separate business and I got lucky again and I was asked to do it in the automotive market. And then I worked with one company and people started moving to other companies and asked me to do it again and again. And I've never intended this to be my career, but I'm getting requests left and right to do the same process excellence, software engineering excellence, work with the framework that I created for Philips at the time by so many different companies. So I just get the best job in the world. I get to solve fresh, interesting problems in software in different contexts and actually see how I make lives better for people.
- [05:16]
Vasco
So, yeah, absolutely. And making lives better for people is what we really like to do here in the Scrum Master Toolbox podcast. I mean, there's even a whole profession dedicated to that in the software world. It's called the Scrum Master, and this is the podcast for them. So maybe Mouly, in those times, you wish you could have probably gotten a podcast like this. I'm just guessing. I don't know, Probably. So when you look at the companies you've worked, we've worked with, pardon me, over all of these years, what do you think is one advice that kind of crystallizes many of the things that you've learned that you could share with our listeners who are right now trying to improve how they build software today?
- [06:03]
Muli
Yeah, I think if I try to generalize my insights, it's a lot around realizing that improvement doesn't mean progress. And the fact that you can improve something doesn't mean it will actually have an impact on your overall results, on your revenue, on your time to market, on your customer satisfaction. And so I think there are too many efforts to improve too Many things that don't really matter. And I think we will discuss this a little bit more when we talk about the AI and the sdlc. But I was working in many companies that were investing a lot of money on improvements that over improving a specific step in the software development life cycle without meaningful impact on the result. And so the ability to tie a specific improvement to what actually means progress for a business, that for me is one critical component that's missing in many transformations.
- [07:11]
Vasco
So when you think about that, okay, so the question of impact is one that we hear a lot about and especially here in the Agile community, that's a dialogue we've been having within the community, but also with our stakeholders for about two decades now, which is when Agile got introduced. So we know it's not an easy question to answer. Right. I bet that as we speak software development lifecycle improvement efforts are suffering from the same problem that Agile adoption suffered. Right. We were told we just want to be faster. Okay, but faster for what? Right? Like just putting more stuff out there is not necessarily going to improve your business. And now here we are, AI adoption. And we'll talk about AI in the SDLC right after this. But here we are, AI adoption and again we're being told we just want to be faster. Well, but you know, shoveling brown stuff faster out the door is not going to make it pink stuff, right?
- [08:14]
Muli
Yeah, yeah. I think at some point everyone works in this industry for long enough has realized that they have to pay lip service to improvement. No one can say I'm happy with what's the current situation. Everyone needs to say I'm improving. Everyone has an improvement plan, an improvement horizon and improvement roadmap. But these improvements, if you ask just one more simple question, how does this improvement improve the business performance? Most people will get stuck because they don't think in those terms of how this improvement ties into the business performance. And if you say I'm only going to do it if it ties back to business performance, then hey, just by that you have probably double the budget because you can cancel half of the projects which are improvement projects, which have nothing but making people feel good that they're doing something, but doesn't change anything in the business.
- [09:24]
Vasco
Yeah, absolutely. And that of course is something we need to figure out. And that's for example, currently one of the things I'm doing with my client, which is to try to define what is the strategic capability you intend to enable with AI adoption. So let's talk about the topic in a recent blog post you wrote that AI can assist across the entire sdlc, pretty much requirements, architecture, co generation, which is where most of the efforts are right now. Testing, documentation, you name it. But the organization needs to identify which phases actually need help before picking the right tools for the job. Well, that sounds rather obvious, but clearly most companies haven't figured it out. Why not? Molly?
- [10:13]
Muli
Well, maybe it sounds obvious when you and I phrase it this way, but if you ask organizations, where across the software development life cycle do you currently suffer the most? Are you blocked because you're late with requirements? Are you blocked because your requirements are immature? Are you blocked because your architecture can't bear the rate of change that's imposed on you? Are you blocked because your testing cannot automate your tests well enough, or because you have instability and reliability issues? You can easily get a list of problems from all of these areas. If you talk to the architect, they will talk about reliability. You talk to the test leader, they will talk about automation. You talk to the peer program manager, they will talk about requirements. But which one is the problem for the organization and which one? If you have just $100,000 to invest in improvement and only one improvement you have to make this year, which one will it be? You mentioned most of the effort goes into code generation and it goes to co generation, not because it's the most effective thing to do, but because it's the easiest thing to replace in the sdlc. It's replacing the one step that's like the easiest to automate and very rarely the actual problem for organizations to becoming more productive.
- [11:56]
Vasco
I totally see that when you think about all of the steps that information needs to go through to be translated into customer value. One of the key things which often leads to a lot bigger problems than how fast are you coding? Is hidden business assumptions about what customer value is. One of the things that we very often talk about is that, hey, we're very good in terms of velocity. We're writing all of these stories or writing all of these story points, as some organizations unfortunately still do. But then we forget to ask, okay, but what's the impact? We delivered 520 features last year. How many of those delivered any dollar or euro of return to the company? And we very often don't know the answer even to that simple question.
- [12:51]
Muli
Yeah, if you look at I really try to generalize the business performance. If you speak to an executive or an engineering leader, they usually want one of three things. They want faster time to market or they want to get higher quality. In other words, less customer issues. Or they want to get better margins, so less cost in their execution. Usually they want some blend of the three. And so the all of their attention on all of the business attention is targeted towards these three goals. Now the challenge is how to optimize a specific step in your software production such that one of these three goals will become better and in a quantifiable way. So you have to be able to say I'm going to improve the way I do code review and that's going to give me 5% less cost of R and D by the end of the year. If you can't say that, why invest in training everyone on a new code review process or tool? Why even interject AI into your code review?
- [14:11]
Vasco
Because AI is not cheap either. I mean I've been looking at some contract proposals for AI providers and we're talking about millions per year for a mid sized organization. This is expensive services.
- [14:25]
Muli
Exactly. And it's going to get more expensive because right now the venture capital market is funding AI, but slowly the enterprises are going to fund AI and so these services will have to justify themselves. So if you cannot tie an investment to an actual improvement in one of these metrics that I've spoken about, then what's the point? Improving for improvement's sake is just, yeah, it's almost meaningless.
- [15:03]
Vasco
So this is something I want to dive into a little bit more. So here's something I've been kind of debating myself with and have had some internal conflicts. When we have data from companies that are out there trying to adopt AI and going as the key word these days is going AI native, we can see that individual developer performance is up. That's easy to believe. And we can see the numbers even from clients that I work with. A survey that I have linked in the show notes is the Pharos AI report which says that the developer PR single developer PR productivity went up 98% with the help of AI tools, but the actual overall delivery at the organizational level dropped by 1.5%. This is data from I believe 2005. So of course things are evolving and getting better all the time. But this is real data from 2005. 2025. Sorry, 2025. So last year at the time of this recording, so more code, same or even worse outcome. How do you help organizations kind of get over that? Like because we need to first start by recognizing that, okay, great, so maybe you help them there as well. But how do you then help them to move forward? Because in this specific case we can see that the organizations surveyed by Farozai were clearly not optimizing the right place of their process and they were still happy and confident. They had indeed optimized something, which they probably didn't.
- [16:44]
Muli
Yeah, more code, less business productivity is exactly what we were discussing earlier, these meaningless improvements. Saying the developers are more productive because they produce more code is like saying a book author is more productive because they write more words. Okay? The unit of work is not the number of lines of code they produce. The unit of work is a piece of code that works, that is tested, that is fully reliable, that meets a customer expectation and eventually generates revenue. Now if you measure that, then you can talk about developer productivity. Looking at the number of PRs or lines of code the developer responds is really meaningless. Especially when the definition of done doesn't take into account the testable, traceable solves a real problem and has a positive customer feedback. So it's easy for the cloud code of the world to show, hey, developers are more productive. Look at how fast the code is spawned. But as you already noted with these surveys, the impact on the business is not, is not there, is not significant. And you asked how this can be solved and the solution is simple, but yet not trivial to do is to understand where in the software development life cycle a specific company is now suffering. Where are you bleeding quality? Where are you losing your revenue, where are you losing your budget? And that is rarely on the side of generating code.
- [18:55]
Vasco
And it can be anywhere, right? Like can be the collection of requirements on the customer side, can be the decision making process on which of those requirements to implement. It can be on the testing side, can be on operations or deployment, it can be anywhere.
- [19:11]
Muli
Yeah, I know we're going to discuss this a little bit later, but if you think about the software development life cycle, which is a fixed flow of activities we have to go through as we make software available to the world. Each phase of this software development life cycle can be your bottleneck, each phase can be your blocker. And in order to understand where to improve, you have to be able to debug this software development life cycle and understand what is keeping you back. It can be how you reuse code, it can be how you use open source or third party code, your policies around dynamic analysis, reliability, non functional requirements. It can be different things for different businesses. But if you just say, well, let's just get 200 million tokens and let the developers create a lot of code quickly, well, there are 35 steps in the SDLC, you have a 1 to 35 chance of being lucky and improving your Productivity. Maybe your problem is code, maybe you'll strike gold. But as the famous quote from Mission Impossible goes, hope is not a strategy. If you want to improve in a structured way, you have to understand where your problems are. And you started by saying this sounds reasonable, but today no one has a really good understanding of. Here's a simple question for you in name an organization or think about an organization you know well and try to answer in your mind the question of for this organization, what is a bigger problem? Requirement stability, Code review or test automation?
- [21:23]
Vasco
Well, with AI in the loop, I know one of those is going to be a much bigger problem than the other two.
- [21:29]
Muli
Which one?
- [21:30]
Vasco
Code review. Can you imagine reviewing ten or hundred times more code with the same people?
- [21:39]
Muli
Yeah. Well, you know that the fda, I'm working a lot with healthcare, but FDA just issued the notice that every AI assisted step has to be guardrailed by human activity. And if you want to include code that's auto generated by AI in a medical device has to be guardrailed by a human reviewer. And if you just think so you have two options. One is generate as much code as you want with cloud code and then do a human code review. But then you understand that there is no actual benefit in generating the code quickly or you offload the entire thing to AI and you say dear AI, generate the code, review it and then let's ship it and hope for the best. Yeah, and the funny thing is that even I think was it last month and Anthropic launched a code review service where they charge $25 per PR to review the code.
- [22:46]
Vasco
The things they will come up with no. And it's like it's more expensive than a highly paid developer.
- [22:53]
Muli
Forget about the business case, but is like the code experts, the one that generate the code are saying we can't guarantee the code is good so you have to buy from us another service that will check that our first service has generated the code.
- [23:13]
Vasco
Sounds like paying for protection to me.
- [23:16]
Muli
Yeah. So if you have the capability to generate code, why won't you generate high quality code? Why won't you generate code that passes code review? So yeah, The way to get great results from AI in the software development life cycle goes in two ways. One is understanding what your specific organization needs help with and second, understand the tool or the capability and what it can actually do for you. And that goes into understanding the capabilities of AI and where it really thrives. And that's I think we're going to discuss later some of the best.
- [24:09]
Vasco
Well, let's talk about that right now because I think it makes perfect sense. So let's dive into the different phases of the sdlc. And you talked about what makes sense to do with AI in any sdlc. So including FDA approved software, power devices, you mentioned AI can provide a lot of value as a code review enhancer. So helping a human doing a code review, defect root cause analysis, test generator. What are the most successful applications of AI you've seen in your clients when it comes to bringing AI into the sdlc?
- [24:47]
Muli
Yeah, yeah. So generally speaking, the best usages would be in fields or areas of the SDLC where there is a lot of data, a lot of data that needs processing and needs some detection of patterns where AI is really, really good. I'll give you a simple example. Think about any organization that you know and look at their Jira. They probably have 10, 20, 100,000 different bugs listed in JIRA from their history. And these bugs tell a story, and they tell a story about patterns of problems, areas of problems, how long it takes to detect a problem, how long it takes to fix a problem, what are typical fixes. And human mind finds it very difficult to detect patterns within this data. But this is where AI is really powerful. If you let AI train on your specific data sets from your specific bugs, asking it very specific questions on how to find patterns in your flow, then you start finding very interesting opportunities for improvement. I can give you one very concrete example. So just recently with one of my clients, there was a claim that the test team is generating a lot of false positive, a lot of bugs that end up not being fixed. And so basically the claim was that the test organization is highly ineffective.
- [26:39]
Vasco
I heard that before.
- [26:41]
Muli
Yeah. So we trained an AI agent to read the defect logs and try to find patterns. And after a while it found that there are more instances of problems where the test team has identified a true deviation from the requirement. This went into a review and the team basically said, well, we don't have time to fix this problem, so the deviation is acceptable. And the bug was closed as a false positive.
- [27:28]
Vasco
What a surprise.
- [27:29]
When have I seen that before?
- [27:31]
Muli
So by doing this analysis with the AI engine, we found that actually the problem was a lot of compromised requirements rather than the assumption on a lot of false positives.
- [27:46]
Vasco
Yeah. Or as the saying goes, it's not a bug, it's a feature.
- [27:49]
Muli
Yeah, it's an acceptable feature. Yeah. And so in the code review the same thing happens. The best teams are trying to learn from every bug that escapes code review, are trying to learn what happened in the code review and how the code review process itself can be improved. Because the code review is one of the early quality gates and it needs to be quite effective. Most of the problems can be detected at the code review, but learning from bugs is quite a difficult process. And this is where again, AI is quite effective. If you try to generate a code review checklist from your manual interpretation of the bugs, that's difficult. If you ask an AI agent to constantly scan incoming defects, rather from customers or from a verification team, doesn't matter. And generate a live, ongoing checklist for you that actually, you know that if you go over these 10 things, you cover the most probable problems that a code review can catch. And so the, the, the AI agent is there to train on the endless stream of defect data and generate for you. If you say, I'm willing to spend half an hour on code review, the AI agent will generate for you the best half an hour you can spend on the code, looking for the best things that, especially if it is augmented
- [29:31]
Vasco
with previous code review learnings that were collected over in some organizations, many years of code review practice.
- [29:40]
Muli
Yeah, exactly, exactly. And so the use of AI on large data sets, like your code review records, like your bugs, like your requirements even, this is where the integrating AI gives you quite a lot of benefit.
- [30:05]
Vasco
Yeah, absolutely. And I think you described very well any process that already has a lot of data that you can mine for patterns and then transform into either guides, prompts for the AI to do certain things or review certain things. And right now, at the time of recording, we know that companies like Anthropic are releasing new models, frontier models, that are able to do security analysis on software out there. And of course they can do that because they have that data behind. And now we can take advantage of that data that is encoded into the models and apply that in our processes. And a very basic thing I remember and mooly, you and I are about the same age, so you probably went through the same process of trying to do human reviews on requirements documents. Right? Like, we actually had hours of meetings just dedicated to reviewing the requirements, because we already knew way back in the 90s that a large amount of problems came from the requirements document, not from the code itself. And I would say that automating document review is a much easier way to see great benefits of AI, because if you give AI both what you learned from the past, but also your strategic objectives, your okrs, then it can correlate those with the requirements that are being processed to get ready for development. And they can find gaps and they can find things that are either contradicting or incoherent or whatever that is. And I know, at least that was my case. I know that humans are very bad at finding this because it requires a very detail oriented, painstaking analysis of everything that was written. Not just the one requirement, but all of the requirements. And that's an example of where I see that AI assisted or even AI driven processes could help us. What other steps in the SDLC have you seen your clients apply AI to with potential benefits?
- [32:16]
Muli
Yeah, so I think the best implementations I've seen to date are around code review, around test generation, whether unit test or component test or functional test. Also a little bit around trying to create requirements, at least assist in requirement creation. I actually have not seen a good case of implementing co generation with AI. And I think the main reason is that the, the part that is missed is that co generation by itself is not the challenge. The developers going through the process of converting the requirements into code. It's actually a thinking process and it creates a lot of discussions with the product managers and a lot of back and forth which help refine the requirement because the requirements are never complete and they always contain contradictions. And when you start trying to implement them, thinking about these requirements, you start seeing those contradictions and you go back to the product guys and you ask for clarifications and this entire exchange now is, is going out the window when you have AI because it, you give it requirements, five minutes later the code
- [33:54]
Vasco
is there with contradictions included.
- [33:57]
Muli
Yeah, of course. Everything you ask for is there. And now, now it's, it's passed on to the testing to find these contradictions if they can. But you basically lose the brightest mind in the. Well, probably some other professions are going to kill me. One of the brightest minds in the, in the chain, the developers, they are basically gone. Okay. And yeah, and then good luck trying to get more revenue, less client escalations or less cost.
- [34:32]
Vasco
Yeah, yeah, absolutely. We should never replace the thinking or as many people are saying in the AI developers community these days, don't outsource the thinking. I think that's a beautiful, beautiful phrase. Molly. We're getting close to the end, but before we go, I have two questions for you. The first one is, what's one resource you'd recommend for someone who wants to dive deeper into this whole idea of AI enabled sdlc?
- [35:05]
Muli
Well, I would invite people to visit our website at bettersoftware.dev. we have a knowledge hub there with a Lot of data on this topic and of course there are quite a lot of other resources. The main thing is try to map your investment to a business KPI. But yeah, you asked about the resource. I think our knowledge hub is quite a good place to start.
- [35:36]
Vasco
Map investment to a business API. It can't be that simple, can it Mouly?
- [35:43]
Muli
But this is business all around the world since the dawn of business, right? You want to do something. If your uncle ran a bicycle repair shop and you say, hey, let's go advertise in the local newspaper that we have a bicycle repair shop here, the first question your uncle would ask is, all right, how many new customers we will get from this ad? And if you said, well, I don't know, but it sounds like a good idea to advertise, you wouldn't get the money for the ad. The business logic hasn't evolved so much. Okay, if you want to do something right, how will this impact your revenue? How will this impact your customer retention or satisfaction? How will this impact your cost of producing the goods? If you can't answer these things, don't invest. Just don't invest.
- [36:42]
Vasco
Absolutely. Absolutely. All right, we're about to end now. Molly, if people want to know more about you yourself, maybe connect and learn about the projects you're involved with, where should they go?
- [36:54]
Muli
Well, I'm Quite active on LinkedIn, so you can look me up on LinkedIn. I usually respond there within a day. And again on bettersoftware.dev you can find the contact form and reach out to me. So I'm very happy to speak to anyone who needs help on that region.
- [37:13]
Vasco
And for all of you who are listening and are already part of our Scrum Master community, make sure to check out the Global Agile Summit 2025 talk by Mooly where he explains how he does this SVLC mapping and evaluation in organizations all around the world from the most demanding organizations as he just shared. Moli, it's been a pleasure. Thank you very much for your generosity with your time and your knowledge.
- [37:39]
Muli
Thank you for hosting me, Vasco.
- [37:43]
Vasco
Alright, I hope you liked this episode, but before you hit next episode, here's the deal. This podcast is powered by people like you. The members who wanted more than just inspiration. They wanted real tools and real connection to people like who are practicing Agile. Every day we're talking access to over 700 hours of agile gold, CTO level strategy talks, summit keynotes, live workshops, E courses, deep dive interviews, books, and if you're into no estimates, we got the pioneers of no estimates in those deep dive interviews as well. Agile Business Intelligence, creating product visions, coaching your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners, plus a private Slack community which is free of all of that AI slop you see everywhere. And of course, without the flame wars. It's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together. So if this podcast has helped you before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the details@scrummastertoolbox.org membership because listening is great. It's important. But doing it together, that's next level. I'll see you in the community. Slack
- [39:35]
we really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes. Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.