B (7:13)

So let's talk about balance, because in the book, you talk about the Atalasoft experience, ignoring tech debt, as you just described, and Trello rewriting too much. How do you, these days, like, with the knowledge you have and the experience you have, how do you navigate that spectrum, how do you find that balance between rewriting too much and not paying down critical tech debt?

C (7:36)

Right. So I started with the idea that how do we make any decisions? Right. And generally cost benefit analysis is an approach. And I'm working off a concept developed by Bob Mesta, who's widely known for his work in product management. He has a concept where he talks about that we have push and pull forces to doing cost benefit analysis. We have the status quo and we have the progress that we want to make, and that there are forces pushing us towards progress and pulling us towards progress, and there are forces pulling us back from progress and even pushing us back from progress. And using that way of. Of categorizing, I came up with eight questions that I ask about tech debt. And they get at those forces. Some of them want us to make progress if we answer them, if we score them high, and some of us want us to back off the tech debt paying if we answer those high.

B (8:43)

It's like a script where we do kind of a self evaluation with these eight questions to get to the point of understanding should we really be investing in paying this tech debt or not. Right. Right.

C (8:53)

Yeah.

B (8:56)

Can you review those? Just High level. Briefly for us.

C (8:59)

Yeah, very briefly. We have some progress we want to make and I ask, if we do this thing, will people outside of our team even understand what we're doing? Is this a visible thing that people can see in the product? Often tech debt projects are just internal, which is fine, but I think if there's some customer benefit in addition to the tech debt fixing, that's great. Another question I ask is, do we have some engineering values that we've accepted on our team and documents and if we pay this tech debt, we're more matching our value. So that's what's pulling us towards the progress. What's pushing us to do these things are the costs of tech debt. So in the financial metaphor of tech debt, we have interest payments and the size of interest payments. I talk about those as being volatility and resistance. So resistance is how big the interest payment is. Like how hard is it to work with this code? And volatility has to do with how much do we encounter it? Do we have to keep changing it? Then we're having a huge interest payment and it's happening often. So those forces altogether, they're pushing us and pulling us towards paying the tech debt, but there's an opposing force that's stopping us from doing this. So one of those things. So I think the biggest thing to stop you from doing, to stop you from being tech debt is the chance of regressions. So if this code works and it's heavily dependent upon by your customers, you have to be careful here. So if you have a big chance of introducing problems into code that doesn't have visible problems right now, it just has this internal tech debt, that's. That, that's something to be wary of and, and not especially if you don't.

B (10:47)

Have any tests around it.

C (10:48)

Right, especially if you don't have any tests, right, especially. There's also how big is this project? You know, like, you know, I know, I know you're the no estimates guy. So, but, but we did have some sense of how big, how much work would it be to fix.

B (11:06)

I actually have a rule of thumb for that, which is always start with the assumption that it will take at least as much as is at it as it has taken in the past. So I have an example that I often give is this one system that had taken 10 years to be written and then the team thought they could rewrite it, rewrite it in, in six months. And what I told the team is, look, if it, if it took 10 years to write, there's so much implicit knowledge there had no tests that you are not even going to take six months to find even half of the things that the people developing the old system have found for sure. So it can't take that much time. So what I usually advocate is a slightly different approach which first scaffold it with tests, then start replacing only one test at a time. If it's an API or a web system, it's easy, right? Replace one page at a time or whatever. But of course not all systems are like this. Some systems are huge monolithic monsters, right?

C (12:06)

Absolutely. And I totally agree. And one of the things that's good about tech debt in terms of estimating is you have a working system that you can use as a basis. Like you're saying, and then another aspect is given you have some idea of the size, how risky is is that estimate? Like how likely is it to be wrong? And then the final question is about uncertainty, which is if I even do this, will it even be better? Sometimes we know and sometimes we just don't know. So when things like that are your problem, like you don't know about, you don't know your problems and you don't know if they're going to get solved, that's when I would more like advocate for a spike, a project spike, a proof of concept to kind of show. Of course, if you have regression problems for sure, tests are what you have to do. I don't think I could imagine going forward if you didn't, if you didn't have.

B (13:02)

One thing that is really interesting is that of course it's never only 0 or 100, right? Like or 0 and 1, there's always a bunch of options in the middle, right. Like when you're looking at a, let's say a big component that is some kind central component that you always need to touch and go back to and whatever, you don't necessarily need to rewrite the whole thing thing you can start rewriting specific aspects like finding the most common flows and try to isolate those. So internal refactoring and so on. So how do you go about deciding in this spectrum from fixing small things to doing some refactoring to writing tests and rewriting the whole thing. How do you go about navigating also these different levels of paying down?

C (13:48)

So I am a not an advocate for big giant rewrites, although some I have had to do them and there's a chapter in my book about one that I think that worked well, a two year one and what it took to do it and the main learning there was keeping them both in parallel the whole time and, and there's, and there's other things to learn about it. But I'm mostly going to advocate, advocate for my piecemeal migrations along the way and reducing the size of the problem over time and you know, even maybe possibly monitoring that, putting up dashboard up, showing it happen. So one of the teams, I wasn't on this team, but one of the teams at Atlassian who are working on the web version of Trello. Trello was Originally written in CoffeeScript and it was cutting edge at the time. But you know, TypeScript came out at, you know, during that time they decided that they would want to migrate all of their code from CoffeeScript to TypeScript and they did that over time and produced there was a bot or somebody would post in our, in the main Slack channel the percent of CoffeeScript going down. It was called the Decaffeinate project and it was a great name and it was, it was kind of a thing that we could all see that it was happening and it took time, but it was done eventually. And that's because those languages are totally interoperable and that happens a lot now these days. Objective C and Swift in my project work totally interoperable. You can do it as a migration over time.

B (15:23)

And like you said, I think that's another thing, right? Like if we do the migration, the risk is always controlled because you can always fall back on the old system. But if you go and rewrite, then the risk is out of control because you only get anything at the end. And of course you're not going to get what you had before because it's not being done. At the same time, I would say.

C (15:44)

In addition to tests, I would also look at runtime observability directed at what you're doing. If you're for example, rewriting all of your angular code is react code or something like that, make sure that those react exceptions or problems are making it into your observability, that they're not just going to the console and the browser, but they're, but they're being aggregated and that you will notice them because they're going to be brand new, right? So like, and they're going to be in code. Your customer expects to have worked already and they won't necessarily have seen it change because like, hopefully you're just, you know, you're presenting them with the same user interface. Just you now have it in a way that you can maintain better.

B (16:33)

In the book, you also Emphasize this idea of small personal practices like fixing mistakes twice, cleaning as you go, or even using onboarding as a chance to clear tech debt. Can you share a concrete story and how those small individual actions really make a big difference for a team's velocity?

C (16:53)

Oh, yeah. So I hope that teams that are hiring have some kind of onboarding process. And maybe a lot of teams say that they would like their, you know, new hires to push code the first day. It's like. And honestly, I think that's hard. But one good thing that you could do that's easy to push right away is running a unit test, right? Like, so it's not going to break the production runtime. It's pretty safe to put in if it has to go through CI, there's many gates, and it's automatically if it's in an area that you expect them to learn because they're going to do more work there. Unit testing is a really great way to learn a system. It's like executable specification that's helping you prove that you understand the system. So I think that kind of thing is a great onboarding and other personal practices that I like. So when I sit down at my desk, the first thing I do is I pay a little tech day, and that could be anything. I'm looking at code, I'm about to change it. Do I even understand it? Am I having some kind of resistance to it? Put in a little helpful comment, maybe a little refactoring, maybe check the coverage, see if I've got tests that are going to help me make progress today. It doesn't take long. It's easy. It gets you warmed up. It gets you going on the rest of what you need to do. Then at the end, before I'm done, if I'm completely done working on a coding session for that day or that time, I break something on purpose. There's this thing called mutation testing where you look at your covered code and you see it's all covered, but then you change something in it. You change a plus to a minus, a less than to a less than equals. You see, do test break. Great, that's what you want. But a lot of times you do this and tests don't break. Well, now you found a problem in your test. I don't fix it, then that's what I'm going to start with next time. It's going to connect the momentum I have now to the momentum of my next test.

B (19:03)

Actually, that's a great point. I've heard this before and write A test, make it fail or check that it fails, because it should fail if you wrote it first and then continue from that tomorrow instead of getting everything green and then coming back and not knowing where to go next. This morning we're doing an AI assisted coding series here on the podcast and we were just talking about how AI needs to be double checked for failing tests. Right. Like if the tests are always passing, just go and change something in the code, the test should fail and if they don't, you already know the AI is not writing the right tests. And what you're saying is this is very important as a way to support this idea of constant improvement in the code, whether it is small internal refactoring or bigger changes. Because without that we're actually flying blind. Right. We don't even know if we're making the progress we thought we were making.

C (20:03)

Correct. Yeah. And I've actually just recently put into my practice, there are automated mutation testers out there. Striker is one that you see in the JavaScript space and I'm using one called mutmut or mut Mute and that's for Python. They'll just find all your covered code, randomly change lines and give you a report of which ones caused test failures and which ones didn't. Then you can just see this whole area. It's covered, tests are running it, but they're not really doing the checks that they should. It gives you an indication.

B (20:41)

So we talked about automated mutation tests. I like the term, I hadn't heard it before. So I'm glad I'm learning something new here. What are some of the rather than personal team level practices that we, that you advocate we should follow to help teams overcome and hopefully eventually defeat the technical debt they're struggling with.

C (21:02)

So I think so one is to have everybody has some kind of style guide, you know, tab spaces, you know, how they do curly braces. But go beyond that. And I think, you know, at the far end, you know, a target architecture, like if you could start over today knowing everything, you know, what would the system look like? You're not, it's not to say that we're going to do that all today, but there's always little chances that people have of, in this pr move it a little towards that. And so having some kind of agreement on the team of what's acceptable in a, a pull request, what patterns we're using now, what are the ones you might see in a large system that's existed over periods of time. Lots of different examples of doing something. But what's what we do now. Okay, like, and what, what do we accept now? A really important thing. And so this is. And I know people do it all different ways, but I really like the way that my engineering manager and PM did it at Atlassian. So I'm going to describe that, which is how do we like budget for this kind of stuff? What they did, what they did is they agreed on a split for the year, a percentage split of how much time would be allocated towards PM LED work and how much time would be allocated towards engineering LED work. And in that engineering led work, the engineers were given autonomy to use that as they saw fit. Part of that was for tech debt, if that's what we wanted to use it for. Part of it could be dev tooling or the various things where things to keep the lights on that the PM might not be aware of, but we had the autonomy to do it within our time constraint. And then the way we actually allocated it was, I'll just give a, for example, a team of 10, you might say 20% of the time would be allocated towards this. So that's two engineers at a time. We would give them time to do that. So we put them on the engineering LED projects for at least a quarter and maybe two where they would have time to do bigger things so they could take on a project that you could never get done, like in a story on a sprint or whatever. But they would be generating many, many, you know, again, stories on their sprints for this one project that they had time to do because they knew they would be given three months to make some kind of impact on this technical debt problem that we had. So I think those are enabling things that teams can do. And then generally I would say what I again, another thing that we did that I also think worked very well is we had a dedicated tech debt quarterly meeting where we would talk about the tech deck backlog, which we treated in the same way that the product manager might treat their. I'm not talking about stuff that's on your. Your ready to work back. Like I'm talking about like all the planning they do before that. So like they might be using annual.

B (24:06)

Or quarterly roadmap planning.

C (24:08)

Right. But before they even talk to developers about it, like so something that they might do in a product like product board or, or JIRA product discovery, like they're doing it not in the norm. Like the backlog is stuff that's already been approved and ready. I think that developers should be running a similar process with their tech debt, a backlog that's out of the backlog. Not it's not approved. We're just talking about it. And I describe and ask those eight questions and kind of get a sense because those eight questions guide you to what you should do. And we talked about some of those things. So like if, if, if it's risky, has regressions and you don't even know if it's going to work, this is when you're going to do a project spike. If it's got big regressions and it's not tested but you're touching it all the time, start putting in some tests. Use the concepts from Michael Feathers book, working effectively with legacy systems to put seams in and get tests into that system before you even think about doing something with it. If you've got something where you're touching it all the time and it's really in your way and it's not that big a problem to fix, well, that's a good one.

B (25:16)

Yeah, yeah, yeah.

C (25:17)

So those questions, like by the way, by scoring them you're kind of giving yourself an indication of the kind of thing that would help because you're trying to reduce the scores that stop you from paying and increase the scores that get you towards paying.

B (25:31)

Check out the book Swimming in Tech Debt before we go, I do have a question though, because we've been talking about individual and team level practices. Of course, leadership in the company has a huge influence on how much attention we pay to tech. That notwithstanding your own example previously at Telesoft. So you talked about the experience you had at Atlassian where you saw how the cto, even though very far away, hard to even talk to, could still have a big influence on what on the field tech leaders decided to do. So how do you help? Especially in larger organizations, the leaders understand that they need to create the right environment for teams to be able to succeed at defeating this tech debt.

C (26:19)

Right. So I'm going to quickly tell a story about the Aladdin cto. One of the things he did in my tenure there is seeing that across all of our products we were having the effects of tech debt and other things on reliability. He did a thing I'm sure he negotiated with the CEO and the head of product they call the five alarm Fire. We stopped all feature development on every product in Atlassian. And he details this on the blog. I'm not this is a publicly known so but and we were told as engineers do whatever it takes to increase reliability in whatever thing you're working on. So I was given a good chunk of time to Think about what were the problems that our customers were having in my section and things I was responsible for and what could I do in that time. And for me, I had worked on the sync system, the offline sync system in the Trello iOS app. And I knew that it wasn't as reliable as we wanted to. And I'm talking about going from three nines to five nines. It was pretty reliable, but we really want, it has to be something that works. We have millions and millions of users, so I'm talking about possibly thousands of errors a day if we don't meet that threshold. So I put in observability right away to just see how bad the problem was because, like, we didn't actually have a handle on what our numbers were. And we could see like, yeah, it was unacceptable. It was, it wasn't as good as we wanted. And so I had been in that I was collecting all the most common errors and I was able to bring that. I reduced it to 25% of the number of errors that were happening every day during that time, which was great. But there are other things that, that they, that, that, that I've seen leadership do. One of the biggest thing is just expressing the constraints. And by that I mean tell the team what your feeling about tech debt is at a high level and what you think generally is the appropriate amount of time to be spent on it and do that with by explaining where products are in the software lifecycle. So I know on a recent, on a recent episode of your show, someone was talking about Steve Blank's book and that's a great book for understanding what it's like to be a startup and product market fit. And during that time you probably are not paying that much tech debt, honestly, because you're throwing things away, you're learning, you're not going to go back and fix problems because you're moving too fast. That's fine. There's another book by Geoffrey Moore, crossing the Chasm and then also dealing with Darwin where he talks about the whole life cycle. There's growth, there's a kind of a leveling off. There's also like a time where there's decline right at the end of more. All those other parts of the product lifecycle you're looking for operational efficiency and tech debt is a good way to get that. And so if the leadership can kind of understand where, where they think they are and express that to the team, the team can find the appropriate projects, but they need that, that guidance. And then once you do that, give them autonomy and let them do what's best within your constraints and then require from them accountability. So give back some kind of dashboard showing what's going on, like that you've made progress. If this thing is some invisible thing that's inside the code base, you still can show something. Like for example in the decaffeinated project, you, you can show the number of copy migrated. Yeah, yeah, yeah, you can, you can show something that your leadership should understand. It may not be understandable or customers might not care about it, but certainly the engineering management leadership will understand it.

B (30:20)

Well, awesome, Lou, it's been a pleasure to have you. So everybody, the book is Swimming in Tech Practical Techniques to keep your team from drowning in its code base. So make sure you check it out and if people want to get in touch with you and know more about the work that you're doing, Lou, where should they go?

C (30:37)

Sure. So I have a website, loufranco.com and that's where I've been blogging on it for more than 20 years. I have a lot of content that you might find interesting about things about agile and Scrum developer productivity. Some of that is in the book, but it's on the blog. I have an email list there too if you want want more information about that and then you can find me on LinkedIn. So Lou Franco on LinkedIn and you know, definitely let me know that you've come from this podcast and I'll be certainly follow you.

B (31:09)

Absolutely. So everybody check it out. Swimming in tech debt. And why not start the conversation on LinkedIn, explore a topic or share a problem with Lew and hear what he's got to write in case if it's LinkedIn or say in case it's this or any other podcast. Lou, thank you very much for being here and for being so generous with your time and your knowledge.

C (31:34)

Thanks Vaska. I really appreciate it.

A (31:39)

All right, I hope you liked this episode. But before you hit next episode, here's the deal. This podcast is powered by people like you, the members who wanted more than just inspiration. They wanted real tools and real connection to people who are practicing agile. Every day we're talking access to over 700 hours of agile gold, CTO level strategy talks, Summit keynotes, live workshops, E courses, Deep dive interviews, books, and if you're into no estimates, we got the pioneers of no estimates in those deep dive interviews as well. Agile business intelligence, creating product visions, coaching your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners plus a private Slack community which is free of all of that AI slop you see everywhere. And of course, without the flame wars, it's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together. So if this podcast has helped you before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the details@scrummaster toolbox.org membership because listening is great, it's important. But doing it together, that's next level. I'll see you in the community.

B (33:29)

Slack we really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes? Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.