← The Stack Overflow Podcast
Loading summary
Transcript47 lines
- [00:00]
A
Foreign. Hello everyone and welcome to the Stack Overflow Podcast, a place to talk all things software and technology. My name is Ryan Donovan. We're going to be talking about the most dangerous shortcuts ever taken. And my guest to discuss that is Tom Totenberg, head of Release automation for LaunchDarkly. Welcome to the show, Tom.
- [00:28]
B
Ryan, thank you so much for having me. And thank you for being willing to talk about these most dangerous shortcuts. I don't know if it was intentional to make a parallel to the Most Dangerous Game, where it's the humans who get hunted here by these shortcuts, but couldn't help but think about the parallel.
- [00:43]
A
So before we get to our topic, we like to get to know our guest. How did you get into software and technology?
- [00:49]
B
Yeah, I think I took a bit of a non traditional path, which is that I got my formal education in music and being a teacher, which surprisingly, you know, there's a lot of crossover. There's a good number of people in software who came from, I'll say, creative arts. And I think that there's a lot of crossover in terms of the types of discipline, daily practice, breaking down big daunting musical pieces into individual notes. Right. And then carrying those same sorts of practices into software. So in my personal life, I've always been a big nerd. I've been building my own gaming computer since I was in high school. And so. So really that was my entry point into technology. And it turns out that a lot of same neural pathways that lend themselves to music also lend themselves to tech and being able to talk about it and communicate with teams and work together in an ensemble. Whether you're rehearsing in an orchestra or trying to build complex enterprise software, a lot of the same skill set is needed.
- [01:44]
A
You're in charge of release automation. I'm sure you've seen some sketchy shortcuts. What are the sketchy shortcuts that you've seen?
- [01:53]
B
To be clear on what my role is, launchdarkly helps with some. Some release automation strategies for our customers. And so what I am helping oversee is partially the product direction and strategy for release automation. How we can standardize the change management process. Right. And put observability guardrails so that we know which metrics we care about, so that as we are releasing and as we are standardizing this practice of exposing something, do we have the signals that we need to automatically respond to that inappropriate way, like pause it, roll it back, that sort of thing. So my perspective on this comes from digging deep into a lot of different organizations change management processes, which is where my, my whole technical background is from, and then also helping them improve that process. So overall there are a lot of themes that we'll be getting into. But one of the consistent themes that I always think about is some of the best engineers out there are fundamentally lazy in that they will take the path of least resistance. You know, if you are measuring something, they will try to gamify that measurement. And as long as the reports look good, as long as it's easy and they're not getting bothered, they're going to take whatever shortcut they want. So yeah, happy to get into specifics there, but it's like water, right? It's going to flow wherever it's easiest to go.
- [03:05]
A
I think there's possibly apocryphal quote attributed to Bill Gates that if you want something done efficiently, give it to a lazy man 100%.
- [03:13]
B
And I appreciate that too. You know, it's the sort of thing where if you can save yourself five seconds of annoying clicks with an hour of upfront work, honestly that's a great quality because that pays off forever, you know, as long as you don't go too, too far overboard with it.
- [03:27]
A
Right. And I think that too far overboard is what we're getting into today. So the shortcuts that you see, where do they happen most of the time?
- [03:36]
B
Some of the fundamental reason, in addition to the sort of structure surrounding the people who are taking the shortcuts, is also the business pressure. There's a trend. We want to go faster and faster and faster. It's not good enough to release twice a year. We have to release either in an agile two week cadence or even, you know, continuous deployment, continuous delivery, continuous release. All of this business pressure then puts on pressure for the technical folks who are supposed to deliver value to their end customers in some way. And okay, as long as we can keep that in mind, then a lot of the time some of the technical details start to get missed. And an example of that that will be sort of exacerbated, I'll say, in the coming years is stuff like homebrewed tooling. This sort of duct tape connector that wasn't necessarily intended to be load bearing, but someone had a solution that they made. And then all of a sudden one of their teammates said, oh, hey, I think I can use that. And then someone else said, oh, I can use that too. And all of a sudden this little duct tape connector is something that multiple people are using and they are getting away with it until it reaches some sort of scale, causes some sort of incident, and bang, now you're in trouble. So a concrete example of that is stuff like configuration management. People have recognized the value of decoupling deployments, which is the technical act of putting code into production or wherever you're deploying it. Right from releasing, which is the business decision to expose new features and functionality to end users. Right. Those are two separate things. Deployment versus release. And so I have seen so many weird, cobbled together, arcane, strange configuration utilities out there that do operate in runtime, that allow you to deploy darkly, but then, oh my gosh, if anything goes off the rails, it's an incident. It's so tough to navigate and figure out the underlying root cause. So that's just one example. There are a lot of those sorts of examples, though, in the process with.
- [05:29]
A
VIVE coding, we're about to enter a golden age of homebrew tooling.
- [05:32]
B
Oh, absolutely, yeah. The cost of entry, the barrier to entry has never been lower. But as with everything, the same sorts of shortcuts exist. And notably, if you're just handing over the vibes to whatever agent you want to code this out, then you are personally more disconnected from the actual code that's making its way out to production.
- [05:52]
A
Right.
- [05:53]
B
Or that you are relying on to safely deliver software so everything gets slower.
- [05:57]
A
As a result, is a sort of like secretly important thing, because that's going to prevent you from leaking API keys, leaking secrets, leaking a whole bunch of stuff into production, right?
- [06:09]
B
Oh, sure. Or even just think about the AWS outage that I'm sorry for anybody you know, still listening where that's in recent memory, but that was a DNS config change, right? And so often what we don't think about is that a lot of these connectors can bypass our otherwise secure sdlc. Right. Or there's governance and reviews and everything put into place. But then if you've got a configuration management tool that's circumventing that, that's still a change. Maybe doesn't have those same controls, the same scrutiny, same sets of eyes, Right?
- [06:39]
A
Right. Does that go through your build process, your test process?
- [06:43]
B
When you've got the duct tape solution, it's not necessarily there anymore. Even a more egregious example that I saw pretty recently, which is by working with customers, I get to see a wide variety of approaches to AI code gen, which some of them are cautious and some of them are saying, no, nothing in production, and some of them are saying, hey, yeah, pop it through the sdlc and, well, you know, we've already got the reviews in place and let's see what happens. So something, something that I saw at a customer was they didn't modify their peer review process. So a human. So Ryan, you could go tell Claude, hey, go generate some code, work on this thing. And then, great, it's going to get a peer review. Ryan, you're a peer, you can review that. So you can tell it to create something. The AI creates the code, you review it. No other human sees that. And so being able to even circumvent that shortcut, even in this forward thinking organization of saying, okay, if it's AI generated code, we need at least two humans because that will make sure that one person isn't going to be the person who shoves something out the door. With the assistance of Claude or Gemini or whatever LLM you're relying on.
- [07:50]
A
Yeah, yeah, I mean, I think people are going to be rediscovering the importance of code review. I think there's a lot of small stuff, rubber stamping, just get this through. But if it's AI generated, it's like, well, what is this? Let's get a couple people to look at this.
- [08:06]
B
Right? Well, and there's of course the old joke about if you want your PR to fail, then make it 10 lines long. If you want it to succeed, make it 10,000 lines long. Because nobody's going to review that. As we know, AI is really good at generating a lot of lines of code. And so that sort of shortcut, that fundamental laziness that we were talking about, if it's taken too far and you start saying, nah, looks good to me, works on my machine, ship it, go right ahead. No amount of automated testing is going to catch every edge case or think as deeply about the end user flow and problem space and actual value that you're trying to deliver to the end customers, like, why are we building this thing? None of that is going to make its way into the context with the same amount of depth as a traditional human developer, at least not yet.
- [08:50]
A
You talk about shortcuts, what is the difference, if any, between a shortcut and tech debt?
- [08:55]
B
So tech debt, I think is a result of shortcuts, right? Shortcuts can absolutely lead to tech debt. And tech debt comes in multiple forms. Right. Whether this is something like wrapping code in some extra debugging, observability, wrapping some code in feature flag or tech deck can also absolutely be the concept of someone trying to get the MVP out the door without thinking about the underlying fundamental platform that maybe should support that feature. So let's say coming up with a new workflow platform or something like that. Right? Okay, great. We have some really easy things, like I need to go from phase one to phase two, but did you create the APIs? Did you make this extensible? Have you thought about eventually putting in branching logic these sorts of things that might happen in the future if we are not taking the time, if we are just vibing out or just, you know, taking the shortcut as a human and saying, yeah, I'll get the MVP out the door without really considering where this could grow in the future. That absolutely is tech debt, because it means that you're going to have to cause yourself and your team more work down the road to refactor everything that you did, which is so much tougher to unwind later, rather than just building it up in the first place.
- [10:10]
A
You know, it sounds like a lot of these are sort of pressures from the business, right? Like, move fast, get it done, get an MVP at the door. And I'm sure every engineer listening has experienced that. Right. Do you have any recommendations for kind of pushing back against that?
- [10:27]
B
When we think about the sdlc, a lot of emphasis tends to get placed on, you know, what's our lead time between we get a feature and we're able to build something, right. And our automated tests, are they succeeding? Do we have any sort of flakiness, like, how long does it take us to deploy or redeploy? But what tends to get missed in that conversation is the planning phase. Now, say what you want about waterfall methodology here, but one thing that it has going for it is that there's a lot of really intentional planning and very thorough, comprehensive planning. Now, I'm not saying that we need to go back to the 90s and early 2000s and, you know, create these, these waterfall and massive Gantt charts or anything, but what we should be doing is taking the time up front to be able to not just define what is this individual thing that we are building, but where does this fit into the overall picture? So. So at least do that so that we're not doing things like duplicating functionality that another team maybe already built. Is this something that could be reused? Great. That is time well spent. If it takes you one and a half times to build this in an extensible way, then that is better than the 2x time from a business perspective that it would have taken to build it twice. The same idea then is about for the feature itself, knowing upfront how Are we going to define success and how are we going to define failure? So what is the risk of failure? What, what types of failure do we care about? Is this, you know, things like performance degradation or is this business conversions that are going to go way down?
- [11:51]
A
Right.
- [11:51]
B
If, if you're processing payments, you got to protect that core flow. You can't introduce an API or something that could introduce any sort of failure rates or massive latency, anything like that. So we need to be able to define the failure modes, but we also should be able to define the success criteria of these individual features. And that in my mind is something that should happen during the planning phase. So we not only have this clear picture, we know where it fits in and then we have a clear picture at that feature level of what's the failure modes, what's the success criteria. And so as we are building this, we're thinking about those eventual pathways and thinking about those eventual signals and how they build up into the greater picture.
- [12:29]
A
I haven't heard anybody rep waterfall in a while. I like the focus on planning and something I've been thinking about is why is it so hard to define what good is? Why is it so hard to define what the success metrics are?
- [12:41]
B
A couple reasons, and this is a big shift in the industry. Reason number one is we have made a big emphasis over the past 10 years, maybe longer at this point to 15 years, into smaller, more nimble, agile teams that have complete ownership. And the complete ownership part, great. I love the idea of flattening responsibilities. And this is one area that AI is 100% changing people's job descriptions over the next few years is making sure that you've got a more comprehensive view on the impact of what you're building. You don't just get to build and not consider the impact. You don't just get to test without thinking about, you know, security considerations and only think about functional testing. So this sort of flattening is great. But the idea of small nimble teams means sometimes you can be shipping your organization chart, which is if you are in a small team, you're going to ship small solutions, you're going to answer small problems and then sometimes you conflict with another team or you had two different ideas of how to build sort of parallel things. That is where suddenly again, more, more tech debt is going to come into play because there is less emphasis on this big overarching plan which going back, the pendulum swings back sometimes there where maybe we overcorrected in a lot of institutions toward these small nimble Teams.
- [13:55]
A
But.
- [13:55]
B
And we have to have some clear top down direction. Even if that means sacrificing a little bit of that individual ownership, it then means that we're coordinating better as a team. There is a balance there. There's no one right answer for any given solution or platform or whatever it is that you're building. But you've got to be able to find that right balance and make sure there is this clear North Star coordinating all of this. That is what enables you to be able to continuously move fast, not eventually run into all the annoying stuff like, oh crap, they built a similar thing over here as we did and now we got to go back to the drawing board. Right.
- [14:30]
A
I've heard of microservices being primarily an organizational feature and not an engineering feature. I remember my last job. First time I saw a microservices chart of everything, the hundred or so services, doing things. There's a lot of services here that don't do anything anymore. Who is owning the whole of this?
- [14:51]
B
Exactly. Yeah. And controversial thing that happened. Elon Musk took over Twitter and then, you know, there were all sorts of instability issues because he like fired a bunch of people and said, you know, slash all of these other services. Honestly, I've never had a Twitter account, I don't actually know, but the website is still going. And the guy, I have no idea what the operational costs look like, but there was some element of him being able to be like, you know what, that doesn't seem useful. And sure, maybe people weren't able to authenticate for a day and they scrambled Again, I am speaking from a place of ignorance on this, but it seems like they slashed a bunch of stuff and that's not entirely a bad thing. The other example that I always think about is this would been early 2000s, the famous Jeff Bezos saying that everything needs to be a platform, everything needs to have clearly defined inputs and outputs. And this is just like an online bookstore at this point, right? Barely an overall retailer. But that sort of focus on interoperability, on clearly defined input outputs, clear measurements, on making sure that as an organization we know how to fit together and we're not just building something in isolation, only for our own team. That sort of focus has real long term benefits. I mean, it eventually became aws, right? And we all know that.
- [16:05]
A
It's interesting because I think it's, you know, built for scale, right? It lets you replace things very easily. And I saw a video a while back that almost every large software company has rebuilt their software from the ground up to get 10, 15% performance increases at some point, planning ahead and being like we're going to have each of these very easy to kill.
- [16:27]
B
Not just easy to kill, but easy to switch. I think a pretty good example of this recently is OpenTelemetry. OpenTelemetry is, if you're not too familiar, it's sort of open standard framework and language around observability. Right. So think metrics, errors, logs, traces and all of this is now a standard that most observability providers have adopted. Right. They can all speak otel. Something that we could do as we're defining our observability strategy then is like should we just use the native methods or can we spend a little bit of time up front, build a wrapper and that will allow us right to then shift and that will allow us to change strategies as new methods are introduced, as new vendors come onto the scene. This will allow us to plug them in if they are filling in functional gaps in the current observability. Make sure you've got that your own internal platform around whatever it is, in this case observability so that you can be nimble with how you are dependent upon these external providers.
- [17:24]
A
Again that's another feature, another bit of engineering work that isn't immediate, that isn't serving the mvp. Yeah, I wonder if you've seen instances where people were successful at implementing that slow as fast. Arguing that other than coming from the top down like you talked about the.
- [17:40]
B
Bezos, there's a couple interesting patterns I think that tend to happen. Number one, you can have sort of central top down buy in and then there might be some platform team. Right. Central ownership. And so I have absolutely seen success with that. I'm thinking of a large banking, financial, retail investing, that sort of customer that I've worked with surprisingly tech forward where they recognize, okay, we've got thousands of applications, we've done mergers, we got to try to onboard people. There is no one size fits all process. However, we are going to have a golden path, a supported set of tools and techniques and concepts and this is a central coe, a center of excellence that will actually be able to support this. Right. And then you can take it and configure it or customize it based on this golden path. But we've got some standard metrics that we're going to measure you on just to make sure everybody's performing well. So for that sort of scale, that sort of size, it is a clear, clear winner to standardize and go slow as fast because it's a forced multiplier on everybody else who's using that. Right. And so they, the mandate and the time and space and resources to actually do the research and actually implement some of this stuff. If you are a smaller team or if you are an individual, tiny branch of a mega corporation conglomerate, you don't have a budget, you don't have funding, you just have your tickets and you got to work on it. Then the other pattern that I've seen emerges is there tends to be this sort of grassroots groundswell of common practices. So the other example I'm thinking of is in the health insurance space, where I have seen, even through lots of different reorgs and shifts and restructurings and whatever other nice words there are for layoffs. I have seen the opposite pattern, which is eventually there are some common practices and standards that tend to emerge and still take someone sufficiently high enough to realize, oh, hey, they're all doing this. They all say that this is the right thing to do. It's time now. We should actually invest in this. And so then that is similar to what I was talking about with the shortcuts before, which is now at this point, you might have common practices, but different vendors, different processes, different accounts that would need to be migrated or merged together. Right? That sort of thing. And so that's a larger scale of the refactoring that we talked about a little bit earlier. But you can eventually still get there eventually. There does have to be centralized ownership, centralized maintenance for this sort of observability platform observability approach or release management approach, deployment approach, testing approach, all of this sort of tooling. There's economies of scale there. But how we get there can be a little different top down or groundswell going up.
- [20:18]
A
It's almost like there's this folk engineering going on where everybody in the bottom is just sort of like getting their piecemeal, their little parts to make the thing. And somebody has to put together the canonical version.
- [20:30]
B
You know, that could be a hackathon. And a principal engineer says, you know what, I'm going to solve this problem. I'm going for distinguished this year. This is my pet project. That could be a thing, or it could just be someone from management and leadership who recognizes, oh, we better do this right, you know, to avoid that duct taping that we talked about earlier.
- [20:48]
A
That is one of the benefits of the top of the food chain, right? Like I used to call it executive laser beam. Right? That's going to make things move.
- [20:56]
B
It could also be personnel changes in leadership too. I've seen a lot of times a leader will come on board and say, you know what, we did this at the last place. I'm going to make a big splash, a big impact in my first year. Let's onboard exactly the same setup as we had at the last place. Consequences be damned at the new place where it might not make as much sense, but yeah, that's so for better or for worse, that laser beam, it has a big impact, right?
- [21:21]
A
That's interesting. We talked about trying to like, convince folks of move slow, to move fast, but like, how do you then resist changes from somebody coming in with a wrecking ball and being like, my way the highway?
- [21:33]
B
This is where I think flattening the responsibilities. And to be clear, by flattening, what I mean is, yes, you can still have specialization. You can be narrow and deep in a particular area in which you have ownership and expertise and time and can never see this. But the expectation, as businesses evolve and become more AI friendly, which is the trend, this will happen. This is the next big wave, right? Like it or not, these are the skills that we're going to have to learn. And so to answer your question, everybody should be able to have a good understanding of the impact of their work and the impact of their processes. So there's a concept called value stream management out there. And the whole idea is, if you're not familiar, there's a really good evocative image that I have in my brain, which is, I think it was a car manufacturer in Germany or something like that from the book that I read. And they were talking about how everybody, if you're sweeping the floors, if you're in accounting, if you are doing maintenance on the machines, if you're in security, whatever it is, everybody can see the product line of the cars that are going out the door. Everybody knows this is why we are here. If I'm in hr, cool. Ultimately, the business is here to deliver cars to our end customers. That is the value that. That is what we are delivering here. And so when you get the question, then of course, hey, this leader is coming in and they want to take a wrecking ball to the safe, stable thing that has been working. If you can provide business metrics on that and say, you know what, we currently have the data, we have a good signal that what this is doing is maintaining great uptime and mttr and our change management is going smoothly. Right. And so we're able to release quickly according to these industry benchmarks. Sometimes if you take a look at yourself, you're like, how did I end up here? Why do I care about business metrics? Who have I become? What happened to the child who then turned into me? But it's important to be able to protect your day in and day out, including from leadership. Because you know what is always a defensible position? Protecting your customers, protecting your end users. So then if they come in and say we want to change this, you can start to have a leadership level conversation and say, hey, this is my concern. If you change it, this could degrade these metrics we're currently measuring here. We don't want to degrade that, let's talk about it. And so then you can have an honest conversation rather than just rolling over and saying, but I don't wanna. That's not a compelling argument.
- [23:55]
A
Yeah, in the initial pitch talked about creating a sort of sustainable release process, right? Getting a point where things are good and can continue that way. But as we've seen, a lot of sort of business pressures are towards getting more. How do you argue for your sustainable release process against the forces of more?
- [24:17]
B
Well, that is where all of the smart automation and surrounding processes need to be built up, right? This is where they go slow to go fast. I think the rubber really meets the road because it is a greater example of setting up a macro on your computer to type in a common string every time you need to type it so you don't have to retype it every time. Right. It is a macro of password managers, that sort of thing, which sure, we got to set it up once, then we can use it forever. And so for release processes it's exactly the same thing. We should be able to have an honest conversation and internally to talk about what are the various different categories of releases that we have, what are the different risk levels that we need to consider, what are the different architectural areas and responsible teams to oversee these. So how can we come up with automation practices not just in terms of how we deliver the software? Who's in wave one, who's in wave two? Are we doing, you know, blue greens? Are we doing like wave canary rollouts? How are we doing this? So for each of these different categories of releases, we should be able to have not just the control strategy, but the measurement strategy. Right. So how do you actually know? How will you confirm that this thing is going well? And there are likely some pretty good reusable metrics. This is actually one of the reasons that I was talking about a centralized observability strategy before is like if there's high level things that you care about is the overall system actually up or not? Which is, you know, we have status pages and we should be able to see if that microservice is responsive or not, that sort of thing. We should be able to reuse these, but they're not going to be the same signals for every single change category. Right. So that, that is where the control and the measurement together forms this concept of automation, which is via control we should be able to determine based on the qualities of the change, which path it's going down. Is this a fast, aggressive, you know, non breaking change? It's something cosmetic. We're updating some wording on the front end. Okay, cool, whatever. Is this something that's a lot more risky because this is a change to a data schema, this is a change to something that involves PII. And we're, we're introducing new APIs for these SOR of things that are a lot riskier, not just technologically but to the business should go through a slower, more controlled, exposition release process. So why should we not be able to release to 1% of our beta users and then measure the effect that it has on that 1% of the beta users? That requires some correlation between who's exposed and how that change is actually impacting the people who are exposed to that change. So there's a lot of interesting stuff that you can do there based on that sort of release strategy to make sure that again the blast radius is contained, that you are not getting woken up at 2am, which is something we all maybe done once and hope to never do again. And so this automation really is those two concepts. The way that I think about it, which is control on the front end, measurement on the back end to make sure that we're not just relying on pre prod automated tests, but then we can actually get the signals that we need to confirm that this is doing the job that is expected of it. And it is not introducing regressions simultaneously so that we can confirm the quality and really validate what's going out the door. That's really what I mean by this standardization that takes a little bit of setup. But then look at all these paved paths we can go down and with the guardrails already on the bumpers in the bowling alley, thinking conceptually rather than on specific practices, specific tooling, I think is going to be more important than ever. Things are changing so fast, we all know that. And so being able to take a step back, think about the concepts, think about the first principles of all this is how we all survive the upcoming AI apocalypse. However we want to call that.
- [28:06]
A
Well, it's that time of the show again where we shout out somebody who came onto Stack Overflow, dropped some knowledge, shared some curiosity, and earned themselves a badge. Today we're shouting out the winner of a great question badge, somebody who asked a question that earned 100 or more points. So congrats to Boris Gorlick for asking Removing handlers from Python's Logging loggers. So if you're curious about that, we'll have it in the show. Notes My name is Ryan Donovan. I edit the blog, host the podcast here at Stack Overflow. If you have comments, concerns, topics to cover, please email me@podcasttackoverflow.com and if you want to reach out to me directly, you can Find me on LinkedIn.
- [28:48]
B
Ryan, thank you so much again for having me. I'm Tom Totenberg from LaunchDarkly. You can find us@launchdarkly.com and we're also, of course, on LinkedIn and Twitter and YouTube, you name it. So come find us and say hi. Thanks again for having me.
- [29:00]
A
All right, thank you for listening, everyone, and we'll talk to you next time. Sam.