← Scrum Master Toolbox Podcast: Agile storytelling from the trenches
Scrum Master Toolbox Podcast: Agile storytelling from the trenches
Swimming in Tech Debt — Practical Techniques to Keep Your Team from Drowning in Its Codebase | Lou Franco
00:33:56
Loading summary
Transcript39 lines
- [00:04]
A
Hey there, agile adventurer, just a quick question.
- [00:07]
B
What if, for the price of a.
- [00:09]
A
Fancy coffee or half a pizza, you could unlock over 700 hours of the best agile content on the planet? That's audio, video, E courses, books, presentations, all that you can think of. But you can also join live calls with world class practitioners and hang out in a flame war free and AI slop clean slack with the sharpest minds in the game. Oh, and yes, you get direct access to me, Vasko, your Scrum Master Toolbox podcast. No, this is not a drill. It's this Scrum Master Toolbox membership. And it's your unfair advantage in the agile world. So if you want to know more, go check out scrummastertoolbox.org membership. That's scrummastertoolbox.org Membership. And check out all the goodies we have for you. Do it now. But if you're not doing it now, let's listen to the podcast.
- [01:12]
B
Hello, everybody. Welcome to this very special bonus episode. And in this episode, we'll be learning how to swim. No, literally, we'll be learning how to swim, but in tech debt. And we have with us to talk about swimming in tech debt, Lou Franco. Hey, Lou, welcome to the show.
- [01:31]
C
Thanks, Vasco. Thank you. I really appreciate you having me on.
- [01:34]
B
Absolutely. So Lou is a veteran software engineer and he's also the author of the book Swimming in Tech Debt. That's what I meant. He has decades of experiences at startup as well as companies like Trello and Atlassian. And he's in both sides of debt, tech debt, as a coder and as a leader. And today he advises teams on engineering practices, helping them to turn messy code bases into momentum. And in this episode we explore his book titled Swimming in Tech Debt. So make sure you check out the link in the show notes. And in this book, he explores how to defeat tech debt. It's not only about the leaders, of course it's about the leaders, but it's also about the teams and our own actions as individuals. So Lou, in your book you open with the image of swimming in tech debt. It's right there on the COVID You can't miss it, where every lap is like pushing against resistance. And you've said that this comes directly from your own journey, from your early coding days to then leadership at Trello and at Talasoft. Can you share how those experiences, some of them, and then how they shaped your view of tech debt and ultimately got you to write this book? Yeah.
- [02:49]
C
Thank you. Thank you for letting me share about that. So my Career is mostly working at startups and then the companies that acquire them. So I did four startups kind of in a row and they all had some kind of exit to a larger company or something bigger. And I spent quite a lot of time in each one, five or six years. And so I got to see code bases evolve over time. And one of the things, one of the formative things that happened to me at a startup called Italisoft, we made digital imaging SDKs for. Net and we were acquired by a large public company called Kofax. And they gave us a two and a half year roadmap that we had to do to get an earn out as part of the deal. And I was the engineering lead, I was the at that. Once we were in Kofax, I was now the head of Talisoft Engineering and I was very focused on getting that roadmap done. Blinders on to almost anything else, including tech debt. And what happened was we had an engineer who was leaving the company and it wasn't. They wanted to move to Seattle across the country and that's great for them. But I did have an exit interview and during that exit interview they really let me have it. About the tech debt in their, in our product and how it was very frustrating to them, not just its existence, but my attitude toward it and that I was basically ignoring it, which I honestly was. And I decided that right after that I would do one on ones with everybody else on the team. And I heard and I told them what I had heard, what their colleague had said and they all let it out. And so I got to finally we can open up really. And so we identified one area. It was an area that we touched a lot, that was part of every product. It was part of the build and installation process. And it was constantly breaking, it was hard to test and it was really just slowing people down and just super annoying. And we could rewrite it with a newer technology that had come out for building this kind of thing. And it was actually not even that hard a project but what it would do is just like eliminate whole classes of problems because it was just a better system. There was Microsoft's new install system. We did it. I noticed this is the thing that really was the formative part. I noticed that we didn't go slower by paying tech debt. We went faster because we were constantly in that code and now we didn't have to run into problems. It was unit testable. This process was easy to test, automated in an automated way. It was just harder to make the kinds of errors that we were making. And so it didn't slow us down, it was giving us momentum. And so that very formative experience. But I also have times when the opposite happened. And one of the things I talk about in the book is a time at Trello. Early on, when I was there, iOS8 came out. So this is 2014, and there was a whole new way of doing navigation coding, the navigation inside of apps. And it was. Unfortunately, it caused crashes sometimes in our app because of this new navigation style. Instead of fixing those crashes, which we could have, it would have been a workaround. It wouldn't have been exactly the right way to do it. But we decided to rethink navigation code in our app, and it took longer than we probably thought it would. And I would say, in general was overkill for what we needed to do for that app. We eventually had to rewrite it again near the end of my time there. So, like five years later. So I'm trying to, in my book, I'm trying to find this balance. When should we pay and when should we.
- [07:14]
B
So let's talk about balance, because in the book, you talk about the Atalasoft experience, ignoring tech debt, as you just described, and Trello rewriting too much. How do you, these days, like, with the knowledge you have and the experience you have, how do you navigate that spectrum, how do you find that balance between rewriting too much and not paying down critical tech debt?
- [07:37]
C
Right. So I started with the idea that how do we make any decisions? Right. And generally cost benefit analysis is an approach. And I'm working off a concept developed by Bob Mesta, who's widely known for his work in product management. He has a concept where he talks about that we have push and pull forces to doing cost benefit analysis. We have the status quo and we have the progress that we want to make, and that there are forces pushing us towards progress and pulling us towards progress, and there are forces pulling us back from progress and even pushing us back from progress. And using that way of. Of categorizing, I came up with eight questions that I ask about tech debt. And they get at those forces. Some of them want us to make progress if we answer them, if we score them high, and some of us want us to back off the tech debt paying if we answer those high.
- [08:44]
B
It's like a script where we do kind of a self evaluation with these eight questions to get to the point of understanding should we really be investing in paying this tech debt or not. Right. Right.
- [08:54]
C
Yeah.
- [08:56]
B
Can you review those? Just High level. Briefly for us.
- [09:00]
C
Yeah, very briefly. We have some progress we want to make and I ask, if we do this thing, will people outside of our team even understand what we're doing? Is this a visible thing that people can see in the product? Often tech debt projects are just internal, which is fine, but I think if there's some customer benefit in addition to the tech debt fixing, that's great. Another question I ask is, do we have some engineering values that we've accepted on our team and documents and if we pay this tech debt, we're more matching our value. So that's what's pulling us towards the progress. What's pushing us to do these things are the costs of tech debt. So in the financial metaphor of tech debt, we have interest payments and the size of interest payments. I talk about those as being volatility and resistance. So resistance is how big the interest payment is. Like how hard is it to work with this code? And volatility has to do with how much do we encounter it? Do we have to keep changing it? Then we're having a huge interest payment and it's happening often. So those forces altogether, they're pushing us and pulling us towards paying the tech debt, but there's an opposing force that's stopping us from doing this. So one of those things. So I think the biggest thing to stop you from doing, to stop you from being tech debt is the chance of regressions. So if this code works and it's heavily dependent upon by your customers, you have to be careful here. So if you have a big chance of introducing problems into code that doesn't have visible problems right now, it just has this internal tech debt, that's. That, that's something to be wary of and, and not especially if you don't.
- [10:48]
B
Have any tests around it.
- [10:49]
C
Right, especially if you don't have any tests, right, especially. There's also how big is this project? You know, like, you know, I know, I know you're the no estimates guy. So, but, but we did have some sense of how big, how much work would it be to fix.
- [11:07]
B
I actually have a rule of thumb for that, which is always start with the assumption that it will take at least as much as is at it as it has taken in the past. So I have an example that I often give is this one system that had taken 10 years to be written and then the team thought they could rewrite it, rewrite it in, in six months. And what I told the team is, look, if it, if it took 10 years to write, there's so much implicit knowledge there had no tests that you are not even going to take six months to find even half of the things that the people developing the old system have found for sure. So it can't take that much time. So what I usually advocate is a slightly different approach which first scaffold it with tests, then start replacing only one test at a time. If it's an API or a web system, it's easy, right? Replace one page at a time or whatever. But of course not all systems are like this. Some systems are huge monolithic monsters, right?
- [12:06]
C
Absolutely. And I totally agree. And one of the things that's good about tech debt in terms of estimating is you have a working system that you can use as a basis. Like you're saying, and then another aspect is given you have some idea of the size, how risky is is that estimate? Like how likely is it to be wrong? And then the final question is about uncertainty, which is if I even do this, will it even be better? Sometimes we know and sometimes we just don't know. So when things like that are your problem, like you don't know about, you don't know your problems and you don't know if they're going to get solved, that's when I would more like advocate for a spike, a project spike, a proof of concept to kind of show. Of course, if you have regression problems for sure, tests are what you have to do. I don't think I could imagine going forward if you didn't, if you didn't have.
- [13:03]
B
One thing that is really interesting is that of course it's never only 0 or 100, right? Like or 0 and 1, there's always a bunch of options in the middle, right. Like when you're looking at a, let's say a big component that is some kind central component that you always need to touch and go back to and whatever, you don't necessarily need to rewrite the whole thing thing you can start rewriting specific aspects like finding the most common flows and try to isolate those. So internal refactoring and so on. So how do you go about deciding in this spectrum from fixing small things to doing some refactoring to writing tests and rewriting the whole thing. How do you go about navigating also these different levels of paying down?
- [13:48]
C
So I am a not an advocate for big giant rewrites, although some I have had to do them and there's a chapter in my book about one that I think that worked well, a two year one and what it took to do it and the main learning there was keeping them both in parallel the whole time and, and there's, and there's other things to learn about it. But I'm mostly going to advocate, advocate for my piecemeal migrations along the way and reducing the size of the problem over time and you know, even maybe possibly monitoring that, putting up dashboard up, showing it happen. So one of the teams, I wasn't on this team, but one of the teams at Atlassian who are working on the web version of Trello. Trello was Originally written in CoffeeScript and it was cutting edge at the time. But you know, TypeScript came out at, you know, during that time they decided that they would want to migrate all of their code from CoffeeScript to TypeScript and they did that over time and produced there was a bot or somebody would post in our, in the main Slack channel the percent of CoffeeScript going down. It was called the Decaffeinate project and it was a great name and it was, it was kind of a thing that we could all see that it was happening and it took time, but it was done eventually. And that's because those languages are totally interoperable and that happens a lot now these days. Objective C and Swift in my project work totally interoperable. You can do it as a migration over time.
- [15:23]
B
And like you said, I think that's another thing, right? Like if we do the migration, the risk is always controlled because you can always fall back on the old system. But if you go and rewrite, then the risk is out of control because you only get anything at the end. And of course you're not going to get what you had before because it's not being done. At the same time, I would say.
- [15:44]
C
In addition to tests, I would also look at runtime observability directed at what you're doing. If you're for example, rewriting all of your angular code is react code or something like that, make sure that those react exceptions or problems are making it into your observability, that they're not just going to the console and the browser, but they're, but they're being aggregated and that you will notice them because they're going to be brand new, right? So like, and they're going to be in code. Your customer expects to have worked already and they won't necessarily have seen it change because like, hopefully you're just, you know, you're presenting them with the same user interface. Just you now have it in a way that you can maintain better.
- [16:34]
B
In the book, you also Emphasize this idea of small personal practices like fixing mistakes twice, cleaning as you go, or even using onboarding as a chance to clear tech debt. Can you share a concrete story and how those small individual actions really make a big difference for a team's velocity?
- [16:53]
C
Oh, yeah. So I hope that teams that are hiring have some kind of onboarding process. And maybe a lot of teams say that they would like their, you know, new hires to push code the first day. It's like. And honestly, I think that's hard. But one good thing that you could do that's easy to push right away is running a unit test, right? Like, so it's not going to break the production runtime. It's pretty safe to put in if it has to go through CI, there's many gates, and it's automatically if it's in an area that you expect them to learn because they're going to do more work there. Unit testing is a really great way to learn a system. It's like executable specification that's helping you prove that you understand the system. So I think that kind of thing is a great onboarding and other personal practices that I like. So when I sit down at my desk, the first thing I do is I pay a little tech day, and that could be anything. I'm looking at code, I'm about to change it. Do I even understand it? Am I having some kind of resistance to it? Put in a little helpful comment, maybe a little refactoring, maybe check the coverage, see if I've got tests that are going to help me make progress today. It doesn't take long. It's easy. It gets you warmed up. It gets you going on the rest of what you need to do. Then at the end, before I'm done, if I'm completely done working on a coding session for that day or that time, I break something on purpose. There's this thing called mutation testing where you look at your covered code and you see it's all covered, but then you change something in it. You change a plus to a minus, a less than to a less than equals. You see, do test break. Great, that's what you want. But a lot of times you do this and tests don't break. Well, now you found a problem in your test. I don't fix it, then that's what I'm going to start with next time. It's going to connect the momentum I have now to the momentum of my next test.
- [19:03]
B
Actually, that's a great point. I've heard this before and write A test, make it fail or check that it fails, because it should fail if you wrote it first and then continue from that tomorrow instead of getting everything green and then coming back and not knowing where to go next. This morning we're doing an AI assisted coding series here on the podcast and we were just talking about how AI needs to be double checked for failing tests. Right. Like if the tests are always passing, just go and change something in the code, the test should fail and if they don't, you already know the AI is not writing the right tests. And what you're saying is this is very important as a way to support this idea of constant improvement in the code, whether it is small internal refactoring or bigger changes. Because without that we're actually flying blind. Right. We don't even know if we're making the progress we thought we were making.
- [20:04]
C
Correct. Yeah. And I've actually just recently put into my practice, there are automated mutation testers out there. Striker is one that you see in the JavaScript space and I'm using one called mutmut or mut Mute and that's for Python. They'll just find all your covered code, randomly change lines and give you a report of which ones caused test failures and which ones didn't. Then you can just see this whole area. It's covered, tests are running it, but they're not really doing the checks that they should. It gives you an indication.
- [20:42]
B
So we talked about automated mutation tests. I like the term, I hadn't heard it before. So I'm glad I'm learning something new here. What are some of the rather than personal team level practices that we, that you advocate we should follow to help teams overcome and hopefully eventually defeat the technical debt they're struggling with.
- [21:03]
C
So I think so one is to have everybody has some kind of style guide, you know, tab spaces, you know, how they do curly braces. But go beyond that. And I think, you know, at the far end, you know, a target architecture, like if you could start over today knowing everything, you know, what would the system look like? You're not, it's not to say that we're going to do that all today, but there's always little chances that people have of, in this pr move it a little towards that. And so having some kind of agreement on the team of what's acceptable in a, a pull request, what patterns we're using now, what are the ones you might see in a large system that's existed over periods of time. Lots of different examples of doing something. But what's what we do now. Okay, like, and what, what do we accept now? A really important thing. And so this is. And I know people do it all different ways, but I really like the way that my engineering manager and PM did it at Atlassian. So I'm going to describe that, which is how do we like budget for this kind of stuff? What they did, what they did is they agreed on a split for the year, a percentage split of how much time would be allocated towards PM LED work and how much time would be allocated towards engineering LED work. And in that engineering led work, the engineers were given autonomy to use that as they saw fit. Part of that was for tech debt, if that's what we wanted to use it for. Part of it could be dev tooling or the various things where things to keep the lights on that the PM might not be aware of, but we had the autonomy to do it within our time constraint. And then the way we actually allocated it was, I'll just give a, for example, a team of 10, you might say 20% of the time would be allocated towards this. So that's two engineers at a time. We would give them time to do that. So we put them on the engineering LED projects for at least a quarter and maybe two where they would have time to do bigger things so they could take on a project that you could never get done, like in a story on a sprint or whatever. But they would be generating many, many, you know, again, stories on their sprints for this one project that they had time to do because they knew they would be given three months to make some kind of impact on this technical debt problem that we had. So I think those are enabling things that teams can do. And then generally I would say what I again, another thing that we did that I also think worked very well is we had a dedicated tech debt quarterly meeting where we would talk about the tech deck backlog, which we treated in the same way that the product manager might treat their. I'm not talking about stuff that's on your. Your ready to work back. Like I'm talking about like all the planning they do before that. So like they might be using annual.
- [24:06]
B
Or quarterly roadmap planning.
- [24:08]
C
Right. But before they even talk to developers about it, like so something that they might do in a product like product board or, or JIRA product discovery, like they're doing it not in the norm. Like the backlog is stuff that's already been approved and ready. I think that developers should be running a similar process with their tech debt, a backlog that's out of the backlog. Not it's not approved. We're just talking about it. And I describe and ask those eight questions and kind of get a sense because those eight questions guide you to what you should do. And we talked about some of those things. So like if, if, if it's risky, has regressions and you don't even know if it's going to work, this is when you're going to do a project spike. If it's got big regressions and it's not tested but you're touching it all the time, start putting in some tests. Use the concepts from Michael Feathers book, working effectively with legacy systems to put seams in and get tests into that system before you even think about doing something with it. If you've got something where you're touching it all the time and it's really in your way and it's not that big a problem to fix, well, that's a good one.
- [25:16]
B
Yeah, yeah, yeah.
- [25:18]
C
So those questions, like by the way, by scoring them you're kind of giving yourself an indication of the kind of thing that would help because you're trying to reduce the scores that stop you from paying and increase the scores that get you towards paying.
- [25:32]
B
Check out the book Swimming in Tech Debt before we go, I do have a question though, because we've been talking about individual and team level practices. Of course, leadership in the company has a huge influence on how much attention we pay to tech. That notwithstanding your own example previously at Telesoft. So you talked about the experience you had at Atlassian where you saw how the cto, even though very far away, hard to even talk to, could still have a big influence on what on the field tech leaders decided to do. So how do you help? Especially in larger organizations, the leaders understand that they need to create the right environment for teams to be able to succeed at defeating this tech debt.
- [26:20]
C
Right. So I'm going to quickly tell a story about the Aladdin cto. One of the things he did in my tenure there is seeing that across all of our products we were having the effects of tech debt and other things on reliability. He did a thing I'm sure he negotiated with the CEO and the head of product they call the five alarm Fire. We stopped all feature development on every product in Atlassian. And he details this on the blog. I'm not this is a publicly known so but and we were told as engineers do whatever it takes to increase reliability in whatever thing you're working on. So I was given a good chunk of time to Think about what were the problems that our customers were having in my section and things I was responsible for and what could I do in that time. And for me, I had worked on the sync system, the offline sync system in the Trello iOS app. And I knew that it wasn't as reliable as we wanted to. And I'm talking about going from three nines to five nines. It was pretty reliable, but we really want, it has to be something that works. We have millions and millions of users, so I'm talking about possibly thousands of errors a day if we don't meet that threshold. So I put in observability right away to just see how bad the problem was because, like, we didn't actually have a handle on what our numbers were. And we could see like, yeah, it was unacceptable. It was, it wasn't as good as we wanted. And so I had been in that I was collecting all the most common errors and I was able to bring that. I reduced it to 25% of the number of errors that were happening every day during that time, which was great. But there are other things that, that they, that, that, that I've seen leadership do. One of the biggest thing is just expressing the constraints. And by that I mean tell the team what your feeling about tech debt is at a high level and what you think generally is the appropriate amount of time to be spent on it and do that with by explaining where products are in the software lifecycle. So I know on a recent, on a recent episode of your show, someone was talking about Steve Blank's book and that's a great book for understanding what it's like to be a startup and product market fit. And during that time you probably are not paying that much tech debt, honestly, because you're throwing things away, you're learning, you're not going to go back and fix problems because you're moving too fast. That's fine. There's another book by Geoffrey Moore, crossing the Chasm and then also dealing with Darwin where he talks about the whole life cycle. There's growth, there's a kind of a leveling off. There's also like a time where there's decline right at the end of more. All those other parts of the product lifecycle you're looking for operational efficiency and tech debt is a good way to get that. And so if the leadership can kind of understand where, where they think they are and express that to the team, the team can find the appropriate projects, but they need that, that guidance. And then once you do that, give them autonomy and let them do what's best within your constraints and then require from them accountability. So give back some kind of dashboard showing what's going on, like that you've made progress. If this thing is some invisible thing that's inside the code base, you still can show something. Like for example in the decaffeinated project, you, you can show the number of copy migrated. Yeah, yeah, yeah, you can, you can show something that your leadership should understand. It may not be understandable or customers might not care about it, but certainly the engineering management leadership will understand it.
- [30:20]
B
Well, awesome, Lou, it's been a pleasure to have you. So everybody, the book is Swimming in Tech Practical Techniques to keep your team from drowning in its code base. So make sure you check it out and if people want to get in touch with you and know more about the work that you're doing, Lou, where should they go?
- [30:37]
C
Sure. So I have a website, loufranco.com and that's where I've been blogging on it for more than 20 years. I have a lot of content that you might find interesting about things about agile and Scrum developer productivity. Some of that is in the book, but it's on the blog. I have an email list there too if you want want more information about that and then you can find me on LinkedIn. So Lou Franco on LinkedIn and you know, definitely let me know that you've come from this podcast and I'll be certainly follow you.
- [31:09]
B
Absolutely. So everybody check it out. Swimming in tech debt. And why not start the conversation on LinkedIn, explore a topic or share a problem with Lew and hear what he's got to write in case if it's LinkedIn or say in case it's this or any other podcast. Lou, thank you very much for being here and for being so generous with your time and your knowledge.
- [31:34]
C
Thanks Vaska. I really appreciate it.
- [31:40]
A
All right, I hope you liked this episode. But before you hit next episode, here's the deal. This podcast is powered by people like you, the members who wanted more than just inspiration. They wanted real tools and real connection to people who are practicing agile. Every day we're talking access to over 700 hours of agile gold, CTO level strategy talks, Summit keynotes, live workshops, E courses, Deep dive interviews, books, and if you're into no estimates, we got the pioneers of no estimates in those deep dive interviews as well. Agile business intelligence, creating product visions, coaching your product owner courses, you name it. You'll get invites to monthly live Q&As with agile pioneers and practitioners plus a private Slack community which is free of all of that AI slop you see everywhere. And of course, without the flame wars, it's a community of practitioners that want to learn and thrive together. It's the best place to connect with community and learn together. So if this podcast has helped you before, imagine what you will get from this podcast membership. So head on over to scrummastertoolbox.org membership and join the community that's shaping the future of Agile. We have so much for you, so check out all the details@scrummaster toolbox.org membership because listening is great, it's important. But doing it together, that's next level. I'll see you in the community.
- [33:30]
B
Slack we really hope you liked our show. And if you did, why not rate this podcast on Stitcher or itunes? Share this podcast and let other Scrum masters know about this valuable resource for their work. Remember that sharing is caring.