← The Pragmatic Engineer
Loading summary
Transcript248 lines
- [00:00]
Gergely Orosz
At Uber. When did you join?
- [00:02]
Charles Axeldine
I joined in 2012. At the time, yeah, the team was about 20 engineers. There was no roadmap, very little process to learn by doing essentially what do you see?
- [00:10]
Gergely Orosz
How is AI already changing software engineering especially for the work that you're doing at Cloud Kitchens?
- [00:15]
Charles Axeldine
We did our own study. We found it was moderately useful. Every single time we have one of those revolution and AI might be the biggest yet, don't get me wrong, but every time we have those revolutions, the press and everyone speaks about replacing engineers, we're still there.
- [00:27]
Gergely Orosz
What are some non obvious recommendations to thrive inside a high growth startup?
- [00:32]
Charles Axeldine
I talked about memorization, using flashcards to memorize stuff. Very powerful. And second, I'm not sure if it's super non obvious but extreme ownership.
- [00:39]
Podcast Narrator
So that means what does it take to thrive at a fast growing company as a software engineer? Charles Axeldine was engineer number 20 at Uber and saw firsthand as the company grew from 20 engineers to more than 2,000 in just five years. He also happened to hire me at Uber and was my first manager at the company. Today Charles works at Cloud Kitchens and has been maintaining the wildly popular GitHub repository Professional Programming in Reading links for 15 years. In this episode we go into what it's like to work inside a rapid growth startup like Uber during its high growth times. How to be a standout software engineer, including tips on how to lead projects efficiently and the importance of shift, stop and lift people around you mindset. Charles has personal productivity tips including why he uses flashcards to memorize things like architecture patterns and data science methodologies. How Charles thinks AI will change software engineering and why migrations is the best use case Cloud Kitchens has found so far for AI coding tools. If you're working at a fast paced startup or plan on working at one and are looking for tactical advice on how to do well in such an environment, this episode is for you. This podcast episode is presented by statsig, the unified platform for flags, analytics, experiments and more. Check out the show notes to learn more about them and our other season sponsor. With that, let's jump in.
- [01:50]
Gergely Orosz
So Charles, welcome to the podcast.
- [01:52]
Charles Axeldine
Thanks Gergey. I'm very happy to be here.
- [01:54]
Gergely Orosz
It is so nice to be back. And this is. This is not the the first time when we were sitting in a situation like this. It was a bit more stressful situation. It was an interview at Uber. You were the hiring manager and I think you were the final interview in a series of interviews and that's how I joined. That's how we Started to work together.
- [02:14]
Charles Axeldine
Yeah. And actually right before you joined, I don't know if you remember that, but I called you to ask you if you could change stack. I think you were hired as a backend engineer. And I asked you to do iOS because this is where we needed. And you said, yeah, let's do it.
- [02:27]
Gergely Orosz
It was Android. It was a very interesting interview situation. I was doing iOS development back then. I did Windows Phone before, and I don't know if you know, but my interview got messed up. I thought I would be doing an iOS interview, and then they put me on a backend track and I did backend before, a few years before. So I was like, all right, let's do it. And apparently I did good enough on it, which was surprising. And then I was supposed to join as a Python engineer, and I didn't do Python before, but obviously Python, you can learn easily. And I was like, okay, let me learn it. I got a Python book, you know, Python 101. And then you emailed me saying, could you do Android? Was this typical of Uber back then to have this kind of hectic or what was the reason it all felt very hectic when I joined?
- [03:14]
Charles Axeldine
Yeah, it was definitely very hectic. And it's interesting because then when you're during those hectic time, like, you feel like this is not normal, and I would much prefer a time where it's much more quiet, but then when it gets quieter, you miss the hectic time. So it was definitely that hectic. It's not the most surprising things that happen when it comes to hiring, but, yeah, good hectic. Right? Like, you're growing super fast, so you have to ask people to be flexible. And I think that's what people look for in startup, right? They want to see things that are very different, where they get exposed to a lot of different things. And that's definitely what you got with a panel for a stack, a background for another stack. And then you get hired and you get asked on your first day. I think it was close to your first day when I asked you.
- [03:58]
Gergely Orosz
It was a week ago.
- [03:59]
Charles Axeldine
Yeah, Android was pretty, pretty hectic.
- [04:02]
Gergely Orosz
And at Uber, when did you join? You were very early.
- [04:05]
Charles Axeldine
Yeah, so I joined in 2012 thanks to the first driver operation who I was working with before. And at the time, yeah, the team was about 20 engineers.
- [04:15]
Gergely Orosz
Like engineer number 20. Ish.
- [04:17]
Charles Axeldine
Yeah, something like that. So definitely super interesting because there was no roadmap, very little process. So definitely discovering a lot of things as you go. And what better way to learn, right? Like when you are at a startup, things are very unstructured and you get exposed to a lot of problem and you make a lot of things up. You learn by doing essentially. And it was a truly amazing experience. I think it was the same for you because you did join and then.
- [04:41]
Gergely Orosz
You joined as a software engineer and then you made your way into engineer management, right?
- [04:45]
Charles Axeldine
Yes, that's right. One year later. Tuan Pham was the CTO at the time, asked me to step in and best career decision ever was really a rewarding experience. And then since then I've moved back and forth between software engineer and engineering manager and I really appreciate doing both. I think I believe in really hands on engineering managers. Yeah.
- [05:08]
Gergely Orosz
And so by being that early you must have worked with Travis Kalanick, the co founder of Uber, right?
- [05:13]
Charles Axeldine
Yes. I actually still have some pretty good emails where when we broke features what was interesting at Uber is that we build a product that help people make a living. And so when there's an incident, it's not only like a feature that is broken, it's also people who are not paid on time. And I remember this event where the payment process for driver failed and this was right before Christmas and so we got forwarded an email from a driver who couldn't purchase gifts for their kids. And I think that shows the responsibility that you have as a software engineer when you own things in production. It's not only features, it's not only code, it's also potentially people's livelihood.
- [05:57]
Gergely Orosz
I remember when I interviewed at Uber, actually you told me something similar and it was the first time I kind of paused of like, oh, I could actually work on something that has real world implications in terms of people relying on it, it being important. And I do remember that whenever we did an incident review, it wasn't just like, I don't know how many ad dollars we lost, which again with some places that's what it is, but it was like how many people's rights could not be there. And then like behind that there's a story, there's a frustration. I mean I've had before when you know, like a driver cancels, for example, I'm in a hurry or it's stuck in a traffic jam, I'm late. And that itself is frustrating enough.
- [06:36]
Charles Axeldine
Yeah. So right now I'm working at Cloud Kitchens, which is another startup that works in the physical world. Right. And today we speak a lot about like AI startups and it's evidently a very important sector. But at the same time, like all of those startups that actually change things in the physical world and have an impact on people's life. I think, I think people should also consider joining them because you can see the result of your work. So that's 1, 2. The physical world is a source of never ending complexity and challenges, right? Like for instance, when it comes to delivering food, you have the optimization across space and time. That is really interesting from a technical standpoint, from an algorithm standpoint, from a system standpoint, from a latency standpoint, right? Like you have a budget of time that is not infinite because the food needs to get delivered. So it's really interesting to get exposed to the physical one. And the other thing I would say is we talk a lot about like startups and virtual products and social networks and their impact on people's lives. That is not always positive. Like we talk a lot about addiction, for instance, and software that is used primarily to optimize processes in the physical world. You can say that it's fundamentally good, right?
- [07:48]
Gergely Orosz
Yeah, yeah. And very different challenges. I feel we are seeing a little bit more of a push where more people are considering and it's a bit more visibility of the software. But you know, one thing that is just not as, I guess, sexy about them is the hyper growth is rarely there. Uber was an exception and Cloud Kitchens might be another exception. So at Cloud Kitchens, how are things similar or different to Uber?
- [08:16]
Charles Axeldine
So you're right, like the time of hyperscale startups is probably behind us. I think you did a lot of articles on this. The end of zero interest rate. What this means is that the model of this hyperscale startup that grows super fast and hire extremely fast is probably behind us and it's probably a good thing. I think the thing we do differently at Cloud Kitchen is we're much smaller in terms of team size and much more focused. And yeah, you're right. It's really interesting to see that there's a bit more humidity, there's less money floating around, which leads to probably better decision. I think at the time of Uber there was probably like a lot of waste in the way we hired, the way we built teams, the way we started projects.
- [09:05]
Podcast Narrator
Actually talking about how we manage projects at Uber touches on the original story of our seasoned sponsor, Linear. The idea for Linear came about when their founders were going through hypergrowth phases at Airbnb, Coinbase and Uber. As you'd expect with real scale, these companies started to slow down. What used to take days started taking weeks, then sometimes even months. Not because people worked less, harder, but because There were a lot more moving parts that needed to be coordinated. As an example, in the early days of Uber, it took a single engineer about five days to integrate, test and ship a new payment method to the app Google Wallet. But years later, it took two months for three engineers on my team to ship and release Google Pay. Because there was so much more planning, Coordination was stakeholders working with other stakeholder teams and the vendors themselves. As teams grow in size, product development gets hit particularly hard. Every team involved in the process uses a different set of tools and workflows. This fragmentation means there's no scalable way to answer what's been committed, what's at risk, who's actually accountable, who are we building this feature for? It's often a total mess. The conventional approach is to compensate for tooling apps with more headcount or more status meetings, but in my experience, it doesn't help much. This is why Linear exists. To give hygrow teams the clarity and coordination they need without the overhead, Linear's founders built a tool they wish they had during those chaotic hypergrowth scaling phases. You can try it yourself at Linear App Pragmatic and see why teams like Ramp and Clay also switched over.
- [10:36]
Gergely Orosz
So let's talk a little bit about that. I do think hyperscaling is behind us because there was this special time where there was so much money on the market and Uber was so good at raising money. You know, like they raised billions for a physical product. For every single round, the valuations kept going higher and higher. We are seeing something similar right now with AI, but it's not in the. In the physical world. Well, you can argue if GPUs are, but I feel that's a little bit different. But let's talk about what it was like to be inside, because I remember when I joined, it was crazy. And one of my first memories of how crazy it was is when I became an engineering manager, I asked you, I was an apprentice manager and you were my manager, and I asked you, how does headcount planning work? And you told me it's like headcount. The way headcount planning should work is we make a plan, we make an ask, and we do it. He's like, in reality, you keep hiring, and when you hit your limit, you get more headcount because it just comes out of this black box and just.
- [11:35]
Podcast Narrator
More headcount comes out of it.
- [11:36]
Gergely Orosz
It's always been like this. And I was like. And you were like, don't worry, it doesn't make sense. Just keep hiring.
- [11:41]
Charles Axeldine
Yeah. Until it makes sense, right? And yeah, clearly there was like, probably a lack of like financial professionalism and being diligent with the company's resources. But I think now, as you've described, right, like with zero interest rate, it's probably much sounder, like financial principles driving like the headcount planning. Um, but yeah, to, to come back to hyperscale, it's really chaos. That's, that's what it is. But good chaos, right? Like chaos where you get exposed to many problems and if you're curious, which I think is a key skill if you want to, or key quality to have if you want to grow, then you get exposed to so many problems at the same time, which means that you have to ramp up on those problems really, really fast. And you get to learn because the best way to learn is to try something, get feedback, and then keep on iterating with a continuous improvement mindset. So that was really fascinating. But you're absolutely right. Like, the hyperscale meant one onboarding per week. So for instance, when you do one engineering onboarding per week, you have to standardize it, right? Like you have to structure it, you have to process it.
- [12:51]
Gergely Orosz
And just in Amsterdam, we were a smaller office, but we had, as you say, on average one or two engineers joining per week in the office. And we started off as, when I joined, it was about 20 engineers. And every, every week we would have one or two new people. And I remember that as a result we actually, we spent a lot of time improving the onboarding process, which is kind of weird to think now because a lot of companies that I talk with today, they don't, they don't care too much about it because they have a new joiner like every second month or something like this. But it lets it. Like we were focusing a lot because we do so much onboarding, obviously we did so much hiring. So we had to optimize around that to batch the days that engineers could be hiring. We had this thing where if there was a crunch time for a project, we might not do interviews for a week. And the recruitment team was really upset about that because their targets were different. Can you explain what you saw, what hyperscale actually meant in terms of the day to day and then how it later eased into just what normal thing is? Because I feel a lot of people have not seen it. They will probably not see it. So it's good for us to like describe what we saw inside.
- [14:05]
Charles Axeldine
Yeah. And by the way, on the hiring side, the irony is usually the team that needs the most New hire is going to be the most impacted by interviewing essentially because you have to interview for your team. At least that's how it's organized at most startups. You remember it, right? Like it's. I don't want to say a never ending stream of incidents, but a lot of incidents.
- [14:26]
Gergely Orosz
Right.
- [14:26]
Charles Axeldine
A lot of incidents where you feel like you are covering the immediate action items that result from the incident but you don't have time to actually fix the fundamental architectural root cause complexity or some of the things. I would say, yeah, incidents definitely.
- [14:42]
Gergely Orosz
And you also encon was terrible.
- [14:45]
Charles Axeldine
So you have those incidents. At the same time, you also need to build new features and build new products and also deliver. Right.
- [14:51]
Gergely Orosz
Well, I remember that we, we had a lot of enthusiasm through the door. So I still think back how we could manage with so much. Like our on call was terrible. Like we were woken up in the middle of the night like almost every night and we tried to fix it, but it was just patches. But I remember that because a lot of new people were joining and people were very excited because it was from the outside. The company was very attractive. They brought so much energy and that energy definitely lasted for a good like four to six months. And we had a never ending stream of people who, you know, just as you were kind of getting tired, someone like, oh cool, let's do this, let's fix it. And there was a mentality of let's fix this. Which on one end was a lot of band aiding thing initially. Later as I think our growth slowed, we started to think about like, okay, how do we fix the system? How do we turn this into a platform? How do we rewrite? Oh, and there are so much migrations.
- [15:46]
Charles Axeldine
Yeah, to a point. Be too many. After 1,000 engineers or 1,000 employees, you don't need a customer. Right. Like you're going to have teams that are going to create migration requests for other teams and the company can operate in a closed circle. Right. Like you can create one. Yeah. The incident is really the best way to learn about your architecture's deficiencies. Right. Like you can look at the architecture on paper and you can see, okay, this component is probably a problem. But the best thing really to know the bottlenecks is to see what breaks. And especially in 2012 and 2013, every Friday evening I would prepare to come back home and then something would break and then it would be firefighting mode. And this is how you learn that oh, redis breaks after that and oh, postgres break after that and oh, we have this queue Here that is not dimensioned correctly or we have this instance or like at the time, yeah, auto scaling wasn't really a thing. So really interesting. A good way to learn essentially about your system efficiency still to that day. Right? That's a great way.
- [16:50]
Gergely Orosz
I'm not sure if it had to do with hyperscaling or not, but we had this rule that when you are deploying a system that seems like, you know, like a medium sized change, go to the Slack channel or the chat channel and let people know that you're doing it. Because I think we're kind of used to this could cause issues. So it's just giving heads up that if you see something going there, just let us know. Again, back then we didn't have that good observability. In fact, we had to build our own observability. There were no vendors that we can onboard. I do think that this high growth, high number of incidents, hard to slow down and we always had to keep shipping. So the trade offs. The reason we couldn't really stop and fix things because I remember we're planning, we're like, okay, we have let's say 10 engineers and we can do three projects at max. And each project has such a high impact. We were talking in the first, the $5200 million of incremental revenue for that team that if we would have stopped to fix the systems, we would have said no to bringing in that much extra revenue, which kind of didn't make sense. So we tried to do both and we prioritized building new stuff.
- [17:59]
Charles Axeldine
And sometimes you have also like regulatory context and other things like there are certain projects that, that you must do and it's not really up to you to, to make that call. But yeah, coming back to the incidents and the fact that you had to announce deployment, this is why I think the most fundamental way to improve this is to decouple the release from the deployment.
- [18:19]
Gergely Orosz
Right.
- [18:20]
Charles Axeldine
First deploy your code and it's behind feature flag so nothing happens. And then you turn on your feature flag just for your user so that you can test and then you can roll out slowly. This approach, very simple approach, works wonders in terms of providing stability because then you get into the habit of making sure that a deploy never breaks in. You first release and test with your user and then because you're the one turning on the feature flag, you are presumably following the feature and monitoring it with the product manager, et cetera. So this is a very simple strategy to ensure that that would have been, I think if we had better used it at Uber we would have saved ourselves a lot of like unnecessary craziness.
- [19:05]
Gergely Orosz
One thing that was really unique about Uber when it came to deploying is for high risk environments like banks. They would typically have different environments. They would have the development environment, they would then deploy to the staging environment. They might have user acceptance, testing and then production. And these are all physical environments that need to deploy it. Uber didn't really do this. Instead we had the concept of tenancies. So we would run the same code everywhere, but we would have a test tenancy, for example, plus feature flags. What do you think about the upsides and downsides of doing either of having the separate stable environment? Especially when we're talking about the physical world, about when you don't want to break things. When breaking things does have a lot of implications. It could be upset customers or money lost or churn or real world impact.
- [19:56]
Charles Axeldine
Yeah. And actually when I left Uber there was no tenancies yet. We were using Dev server. I don't know if you remember that time where every engineer would have their own development environment where you would install the whole stack on one machine. Evidently the matching was oversized for what we were doing. Is it? So we were installing the whole stack of Uber or at least most of the stack of Uber and then that's what you would use to test, end to end. And what's great is you could share it, you could go crazy with it. But yeah, I think we missed having a pre prod or staging environment. I think you need both. Right? Like you need a staging, you need a way to have or at least the three of them, right? Like you need a way to quickly deploy one instance of your service and then route traffic with it. With tenancies, you need a staging environment where you can go crazy and probably some kind of like pre produced where it's much more stable and you open an incident if something is broken. The idea being the earlier you detect a problem the better because it speeds up the process. Right? Like if you detect an incident at production stage, you're going to waste hours rolling back, maybe fixing forward if you don't have a good rollback story. So the earlier you detect, the better.
- [21:07]
Gergely Orosz
I remember that we had an incident where someone on social media posted that something is broken at Uber. And I remember you were upset and you were upset that we didn't catch it, that our monitoring did not catch it and that someone had to post on social media. I remember, you might not remember what you told us. Like this is the ultimate failure of us needing to learn from a customer that our system is broken. And instead of us, like, like, like, we should have had a system flag it. And, you know, if we missed it, that would have been on us. But, and this still stuck with me, this lesson of, you were so mad not because of the incident, but the fact that we did not have the means. We were basically flying blind and someone exposed us. I mean, it was kind of a good thing. But this stuck with me of, like, since then, I remember, like, okay, how can we make sure that we will know and we will not have to have someone tell us, like, okay, this is broken.
- [22:04]
Charles Axeldine
Yeah. Frankly, to this day, I don't think even on my current team we still have to work on those things. Right. There are three metrics at Cloud Kitchen that we always look at for any incident. Time to detection, time to mitigation, and time to resolution. And the most important part for me is time to mitigation. It means how much time does it take us to bring the system back to a state where the customer doesn't see the impact. Right. Resolution goes after. And if you waste like one week of time because you did not detect the issue, there you go. This is what you need to fix. And at scale, especially at Uberscale, being able to detect is much easier because you have such a high volume that if something breaks, you must have some kind of monitoring that will detect it.
- [22:51]
Gergely Orosz
One thing with hyperscaling is things grow so fast. You keep hiring people, but you're hiring people a lot slower than what your growth is in terms of customers, features, all those things. Automation becomes really, really important. You've always been a huge fan of automation. I remember at Uber, you kind of built your scripts to automate this or that. What have you learned about automation? Of how it works, how it doesn't work? Maybe some things are not as obvious about it.
- [23:18]
Charles Axeldine
The most recent thing I learned about this is this article I read about the ironies of automation, which is a pretty old article, but it's fascinating and it talks about two things that I've definitely seen and I've made a lot of mistakes in this area. One, sometimes when we automate, we replace user error with automation designer error. Right? Like, we are lacking context. We don't understand how the software or the process is used. So when we automate, we actually put in the wrong automation. That's one. Two, because you cannot automate everything, you usually automate the simplest things and you leave the user in charge of the most complex stuff. So that's the irony here is like automation sometimes fails. So what can you do to prevent that? The simplest thing is observe and get a lot of deep understanding of what the process is. You have to do it manually. Right. Like we say startups should do things that don't scale first. So do the things manually first. This will give you the business context, product context, operational context to then automate correctly. And then the second thing is transparency. Right. So many automation, particularly in the developer experience world, automate so much, you don't see anything and then when something breaks, you are left to debug. I'm sure it happened to you at Uber with some of the developer experience products.
- [24:39]
Gergely Orosz
Yeah. And when you said to do things manually at Uber, we had this really interesting thing called sanity tests. Sanity tests were we were in charge of our team owned payments. So everything in the Uber app there were like 12 different ways to pay from credit cards, Apple pay, Android pay, and there's a lot of like regional things. Like in India there was paytm and a bunch of others. And we would just need, every week we would need to make sure that the payment would work. So we would go there, we would order an Uber and like see that the payment went through and we would actually do this manually. Backend was sometimes mocked up, but every time we would go through it and when we did it long enough we're like this is. And we needed to do it manually for a few reasons. One of them it was just hard to automate because we had to talk with the real payment providers and a lot of them that did not offer APIs to do this. There was also fraud detection, but we kept doing manually and after a while it became really, really painful. And then we figured out how can we automate this to some extent. And then we actually started to make progress. But we started with just this thing and we did it for I think many weeks. And in the end we got so tired and frustrated that we were looking for creative ways to get out of it.
- [25:51]
Charles Axeldine
Yeah, I don't know if you remember, but we had to ping an engineer in India to share with us the code that was sent via sms. It was like so inefficient. I mean the end to end test topic is a fascinating topic. I think there is something to be said also about automating them too much because the more first like payment and login flows typically are designed against automation. So I think that's one of the reasons we couldn't really automate them for a long time is because specifically designed to counter that. And then the Other thing is when you automate too much those end to end tests, then you get a lot of flakiness because you're not only testing your flow, but in the case of this SMS based payment flow in India, you're also testing the quality of the SMS reception, any latency there. You're testing many different flows from the banks that you don't control. So if those tests are constantly failing, then you get flaky test and fakey test gets ignored.
- [26:53]
Gergely Orosz
So one interesting challenge during the hyperscaling time was hiring. And again most companies will not be in hyperscale mode, but every now and then you need to hire really, really fast. This might be because you get a big funding round or further reasons. What did you learn about what works? When you need to hire efficiently and you need to hire a lot of people, oftentimes experienced people, you look at.
- [27:15]
Charles Axeldine
The whole process, right? Like you look at every single step at the process as a funnel, as a pipeline and you see where you do you have a leaky bucket. So a metric standpoint, it's very important to have that. The other thing is you build a partnership with a recruiter. So many people don't do that. You have to build this partnership so that you have a feedback loop where your recruiter knows what you're looking for and make sure that those candidates are sourced, are front screen from a recruiter standpoint and then get through the process. And then the last thing I would say is like every single step of the process you can, you have a continuous improvement mindset. One thing you remember we did at Tuber, we do it at Cloud Kitchens as well is pairing interviewers. You have two interviewers and this way they can give each other feedback, they can train each other. There is only so much you can do when it comes to interviewing training. And so to pair interviewers you can share with feedback between the two.
- [28:10]
Gergely Orosz
Yeah, I remember the two things that we did and I don't think many places still do it. It's like first we had a weekly catch up with the recruiter. So engineer managers and recruiters, we would do it. It's a bit like a team retrospective and they would talk about things like what are their sourcing strategies like? Oh, what kind of cohorts are they targeting? LinkedIn has this new feature that I can and they will share tips and we're like, oh, pretty cool. Or like talking about what kind of companies they're thinking of targeting and we'd be like, oh, what about this one? I just heard that there I Don't know. There's some negative news coming from there. Maybe people will be interested in of joining from there. And I've never seen that before. And it was really good. And when we started hiring, we would just sit down with a recruiter, like, spend an hour talking through. Here's what I'm thinking. Here's what this person would do. Here's what I think a good hire would be. But the recruiters bring a lot of expertise. So they would ask questions like, okay, like, for example. And a lot of clarifying questions like, I get so annoyed as an engineer when a recruiter emails you and said, like, I'm looking for five years of C experience, because if you've done Java, Python, whatever, you can pick up C. But we talked this through with the recruiter, so they asked, like, what is your tech stack? And we said, okay, Python, a little bit of Node js. Back then we were introducing Go. So they were like, okay, so we need people with expertise in these languages. We're like, no, no. As long as they did some language, that's okay. In fact, we don't care about the language that much. And they're like, oh, okay. And because of this, we helped the recruiter find better people, or not reject people who could have actually worked out 100%.
- [29:45]
Charles Axeldine
This relationship between the recruiter and the engineering manager is critical. And actually something that came from Tuan Fan, which you should have also on your podcast. He told me that every morning he would come and go and check out with the recruiters and say, hey, what are some of the interesting profiles you saw this week? And it's a great way to have this feedback loop. The recruiter bring a lot of value and have their specific skill set and you know who you're looking for, you know who your team is missing. So this is like, if you don't have this tight feedback loop, it's going to take a lot of time to hire.
- [30:22]
Gergely Orosz
Yeah. And then we had this interesting thing with the pairing. As you mentioned, with interviewers, we always had a primary interviewer and a secondary, and the primary one would lead with the questions. The secondary would sometimes step in and you can discuss in advance. And we had this concept of if someone signed off as an interviewer or not, and when they were not yet signed off, they might lead the interview, but then the secondary was a lot more experienced. They would give them feedback afterwards. So after the interview they would come and say, like, okay, here's some feedback on. You could have let the candidate talk, talk, talk a bit. More. It was great that you jumped in and gave tips and so on. And we tried to. Again, this was something I've not really seen before. Most engineers, I think, are just not really good at interviewing and they rarely get feedback. And we're trying to get this feedback going, set expectations. And we also did some interviewer training, but as you say, it's hard to do interviewer training.
- [31:16]
Charles Axeldine
Yeah. So we have the same process right now. So you shadow first the primary interviewer and then you reverse shadow, meaning you drive the interview and then the other experience interviewer will give you feedback. Yeah, you're absolutely right. Like, just because you're a great engineer doesn't mean you're immediately a great interviewer. You can become a great interviewer with training and with on the job feedback. So that's why this dynamic is critical.
- [31:40]
Gergely Orosz
So how do you know if you're in hyperscale? Hypercoat. Sorry. And how might you know that you're getting out of it? Like, what, what? Again, you've gone through this at Uber, at Cloud Kitchens, you might have had some phases for it. What are kind of signs that you're like, okay, or this is just really intense growth and I, I need to operate differently in, in this mode.
- [32:01]
Charles Axeldine
Yeah. I would say it's a bit what we talked about in the beginning. Right. Like, so a lot of hiring and then things breaking left and right without giving you the time to fix, like the fundamental root cause. And I would say this is probably the best, the best sign that you're getting out of this hyperscale is when you feel like, okay, now I have time to actually look at this thing holistically and fix and build a system that I'm pretty happy with. Not necessarily a second system from scratch. It's not always a good idea or rarely a good idea, but when you get into that territory, that works.
- [32:35]
Podcast Narrator
Speaking of thriving in hypergrowth, one of the biggest challenges is maintaining release velocity while avoiding outages as you scale. From my personal experience at Uber, this is hard to do. Most engineering teams think they have to pick one shift fast and accept higher risk or, or slow down releases to stay safe. But here's the thing. Hyperco teams at companies like Notion, Brexit and Atlassian don't have to sacrifice speed for safety. They use the same toolkit that powers Meta and Uber's experimentation systems. And now it's available to every engineering team. They all use statsig. Statsig is our presenting partner for the season, and they built a complete data platform that lets you ship features and Measure exactly what each change does to your key metrics. This is the same infrastructure that growth engineers use to run hundreds of A B tests per year. Every feature release shows you exactly how it moves. Conversion, retention, revenue, whatever metrics you care about. Built in analytics, feature flags, session replays, the full toolkit. No switching between different tools or trying to stitch together your own measurement system. The gradual rollout approach is crucial here. Start with 1% of users, see how.
- [33:40]
Gergely Orosz
It affects your metrics.
- [33:41]
Podcast Narrator
Then progressively roll out to more users if something goes wrong. Instant rollback. If it's working, you can confidently scale it up. The validation here is incredible. OpenAI was such a heavy user of Static that they decided to acquire the company.
- [33:54]
Gergely Orosz
Talk about product market fit.
- [33:56]
Podcast Narrator
Static has a generous free tier to get started and pro pricing for teams starts at $150 per month. To learn more and get a 30 day enterprise trial, go to statistic.compragmatic what.
- [34:08]
Gergely Orosz
Can you do when you're feeling really overloaded? I remember and you were my manager but there was a time where you looked pretty burnt out. We had performance season and a manager left the company and suddenly their reports also reported to you. You had something 30 reports and you had to do 30 performance check in conversations. I remember that when you did the last one, the few of us more experienced engineers we knew like we saw your calendar, it was full and when you came out we were there. We're kind of clapping that you're done because it took like two weeks. And I remember at the time like it was so you had 30 reports you were expected to get, still get stuff done. I think you were still doing some, some hands on work and like you could tell that, that usually you're pretty chipper usually but back then you are not chipper. What did you do to get out of this situation or to, or to push through it?
- [35:06]
Charles Axeldine
Yeah, I would say there's two things. There's one personal productivity. It's a topic I've always been super passionate about so I've always invested a lot. Like the first book I read on this topic was Getting things done by Alan, which is a pretty good. I think it's also a section in your, in your book by the way. It's a critical skill. Right? Like the, the earlier you invest in your product personal productivity, the more dividend it's going to pay. And I really believe in compound interest rate. Like you invested this and then the second thing, it's a principle that works wonder everywhere. It's divide conquer. So yeah you're Right. Like I remember that time you have 30 reports, you cannot do everything. So now you're going to have to find people who can take over from you some of those topics. So I remember asking you to tech lead a project, for instance, and it's a great way because you give responsibilities to people, you help them stretch outside of their comfort zone and you're saving time. So it's a win win thing.
- [36:03]
Gergely Orosz
I remember that you did it and what I liked about it is you were very explicit about the ask. You were like, I, I need you to. Yeah. For example, lead this project and I don't have as much time to spend on it. I'm expecting to check in with you once a week. If you need something more before that, let me know. And, and then I think you also want more specifically, I like you to make a plan on how you're going to do this. Let's check in with it. And it was, yeah, so it was a way for you to spend less time, give more responsibility. And actually people liked it because people were growing professionally and they also understood the reason of why this added responsibility is going to them. And if you're smart, which I think most people were, they understood like, this is a good thing because with more responsibility means that I now have more opportunities to prove myself. I can now grow in a direction either to be a lead or if they wanted to be later, a manager, or into the staff direction and so on.
- [36:58]
Charles Axeldine
Yeah, this divide and conquer is very similar to what we do with software. Right. Like we create an abstraction, we give it responsibilities, we design an API that is clean and then evidently some details might leak, some implementation details might leak. So with a project, you're giving the responsibility of a project to somebody, but some issues might require your involvement and that's all fine. But then at the end of the day, that's still what you expect from the person who takeover.
- [37:25]
Gergely Orosz
Right.
- [37:26]
Charles Axeldine
Like you're expecting them to drive and to somewhat hide some of the implementation detail from you. And yeah, so it provides growth opportunities, that's for sure. One of my manager told me, like one of the most powerful words you can or expression you can tell somebody is, I need your help. Right. Like this is very powerful, very simple. Right. I need your help with this thing. And the employee feels also empowered to make decisions.
- [37:48]
Gergely Orosz
Yeah. And by the way, I think you can do this not just as a manager, but also when you're a more experienced person on the team and you have someone who's less experienced. And it does work magic, I think it does create a sense of responsibility. I think we also tend to sometimes forget that as soon as you hire someone, even if they're a brand new grad, these are fully capable people. Sometimes I think back on how in the industry, sometimes we have the tendency to kind of over baby the junior engineers. Like, oh, this person has no experience. Well, if you think back of the people who built Facebook into the trillion dollar companies Today, there were 19 and 20, like dropped out of college. So all I'm saying is like people have, especially the people who go through a pretty like high standard hiring process, they actually have a lot of capability. And I sometimes need to remind myself of that when I was a manager or a tech lead.
- [38:43]
Charles Axeldine
Yeah, plus the best way to learn is to make mistake. So you know, if you want to invest in your people, you have to let them make their own mistake.
- [38:50]
Gergely Orosz
Yuki is still right. One of your most popular GitHub repositories, which keeps going viral whenever I mention it, it's called how to be a Professional Engineer. It has a few tens of thousands of stars and it's this collection of really, really good reading resources. How did that start and how are you updating it? I understand that you start to do it for yourself initially and in fact you already did it when we were at Uber and you forwarded it to people, you were like, oh, check this out if you'd like to become a better engineer.
- [39:20]
Charles Axeldine
Yeah, I think the way it started is I wanted to automate the process of giving people feedback and sending them to certain good articles or buyback to automation.
- [39:29]
Gergely Orosz
Yeah.
- [39:30]
Charles Axeldine
So I started compiling this list of topics and I put what I considered like the classic article on this topic. So really like articles that have really changed people's mind that, that are really influential and kind of, I've driven like a lot of engineering practices and then I started in doing this. So one thing, speaking of personal productivity that you can also invest in as an engineer is have a good process for keeping yourself up to date with what's going on in the industry. And so I try to read, I mean total one hour per day fiction work as well. We can talk about that later. But I try to read like a good bunch of engineering articles every day. And if I find a new classic article, I add it to the repository. And I've been doing that for the past 15 years, I think something like that.
- [40:18]
Gergely Orosz
How do you find where to read? What are your sources?
- [40:21]
Charles Axeldine
Yeah, so the Main 1 is Akar News.
- [40:24]
Gergely Orosz
It's a good one, isn't it? Yeah, it's still amazing.
- [40:26]
Charles Axeldine
I can use with. I received like an RSS feed with a 10 top article of the day. Plus I check it because sometimes you have like some diamonds that don't necessarily make it to that top 10 list. There are some really good newsletter. The bytes one for front end engineer. The it's hilarious. On top of this like every time I have a chuckle because it's just so well written. I also follow newsletter about the programming language that I use. So there's Python, Java for ck. Yeah, a lot of the software tech lead is really good as well.
- [41:04]
Gergely Orosz
You are an engineer yourself, you manage a lot of pretty good engineers. Can we talk about a story of, of a standout software engineer and what made this person stand out?
- [41:15]
Charles Axeldine
Yeah, there are so many good stories. I'm not sure I want to single out somebody in particular, but some of the traits of like an archetypally typically good software engineer. I would say there's a couple things. There's one shipping, right. So I've written three or four carrier ladders and a lot of carrier ladders in the industry. They after senior, they over focus on meta work, reviewing RFCs, attending meetings, influencing that, strategizing this. It's really difficult to understand what's going on. So the first quality, and we try to put that in the cloud kitchen's carrier ladder is a focus on building shipping value, being creative, being an expert in your programming language, in system architecture. So that's one shipping is really key.
- [42:05]
Gergely Orosz
Right. So even at the above senior like the staff engineer level, like you still place focus on, you still need to ship things, you need to get into production.
- [42:13]
Charles Axeldine
Absolutely. And we expect for instance staff engineer to really find creative ways to speed up execution or achieve a 10x improvement in quality. And that's, that's what we expect, not only reviewing RFCs or attending meetings. Right. So. So that's one shipping and lifting.
- [42:30]
Gergely Orosz
Right.
- [42:30]
Charles Axeldine
Lifting people around you. This is pretty cool, right? Like we are in knowledge, we are knowledge workers essentially. Right? Yeah. So you have to train people around you. You have to give a hand, you have to help, you have to have a good attitude. I think the best engineers that I've worked with have this amazing attitude of ownership, of taking a problem and not stopping at team boundaries. For instance a couple stories here, but like a software engineer was like identifying a problem and they're like a mobile engineer and then they go into the API gateway, they see the problem is not here and then they keep going, they go to the backend and they actually make the fix in the backend even though they're a mobile engineer. Right. And by the way the amazing thing is in this age of AI you can really bootstrap that with AI. Right.
- [43:16]
Gergely Orosz
I feel there's starting to be no real excuses because before AI you could say it is hard to onboard. It's hard to get to know it. It takes time and effort and I mean it still takes time and effort to become an expert. But as you say like to unblock yourself like at an age in time when a product manager can open a poll request that might be accepted. I think there's no. I feel engineers are necessarily becoming full stack with an expertise in their homestuck.
- [43:41]
Charles Axeldine
Martin Foler posted a brilliant article I think about the expert generalist. It's funny because he mentioned that this T shaped person that we often talked about like this person who is an expert in one thing and one thing only and then broad on it comes from the Valve Software Handbook. But I've never seen that in real life and he made me realize that actually. Right. What you're looking for is more like a rake, somebody who can go deep in a lot of different areas. And with AI you're absolutely right. Like for instance one of the best use case for AI I have seen so far for me is like there's a problem with this feature. Find me where the code is and then I read the code and I understand or maybe I need to understand how much time it's going to take to fix this or that or to implement this or that. Find me the code where it is. And we use a monorepo at Cloud Kitchen and that helps. Right. Everything is in one spot. So you can ask those kind of questions and get this, this kind of.
- [44:40]
Gergely Orosz
Answer and so what other characteristics. So shipping ownership.
- [44:46]
Charles Axeldine
Yes. Lift. So ship and lift. Yes. And I will add there are so many things. Right. But I will add one thing. Humor. Taking yourself lightly is really important. I mean it's evidently not limited to to solid engineer but I think humility and being able to be to self deprecate it's pretty.
- [45:06]
Gergely Orosz
I like how you put it because I feel if you have self deprecation humor a little bit it's hard to have a big ego because I feel that's the thing that gets in the way of some engineers who are really stand out and really smart but they become insufferable if they keep putting their ego ahead of them and it goes against everything. They're not going to lift others if they're. If they don't have that learning mindset.
- [45:30]
Charles Axeldine
Yeah. And also sometimes you have those debates, right? So you have Those debates on RFCs, on code, et cetera and you see some tension and some conflicts. You more diffuse these things. Right. So it's a really powerful move. Yeah. There are so many other things like structure and method. I think a good solid engineer has a method in place for fixing problems. It starts with observability metrics. Right. A good story here would be we are in an incident, you're debugging, everybody thinks like the problem is in System A. And then you have like a really creative engineer who says, actually it's here and we are all looking in the wrong direction. And here's a super quick way to mitigate this problem. This is like also solid engineering. It goes back to the shipping thing. You can only do that if you keep close to the code.
- [46:23]
Gergely Orosz
Now, in this repository you have a lot of really good articles and I love how it starts that I would like this repository to not be overloaded. So I will limit to the most important and impactful documents. And again, about 10 years ago when you sent it to me, it was a lot smaller, but now it is becoming a little bit overbearing. Just there's so much reading. Right. Again, I really appreciate that every one of them is worth reading. What is your take on the importance of reading versus doing? Because if you read all that article, I'm sure it'll help you to some extent. But let's be real, if reading can make you a great engineer, then anyone could just read the whole thing. How do you think about balancing this thing? Reading, doing?
- [47:09]
Charles Axeldine
Yeah, absolutely. So evidently this repository is not designed for being read. Like head to cover. Right? Like cover to cover. You are confronted with a topic and what happens is usually during the day you're in doing mode. Right. So you need to get to results. I think people will start looking at you sideways if you start taking a book and reading a book on the job. Like people usually don't really see that super well. So what you do is during the day you try to make it work and you're going to try many, many things. You might choose AI to unblock yourself and then what you can do outside of work time is read a book about the fundamentals of the topic you're confronted with. Because when you're trying to make things work, sometimes what you're missing is like the fundamental of this topic and if you add them, it would unblock you and considerably speed up your process. So that's why Reading and doing, they go hand in hand.
- [48:04]
Gergely Orosz
Right.
- [48:05]
Charles Axeldine
And you have to do both.
- [48:06]
Gergely Orosz
Yeah. So you're saying a good way is to read related to the problem area that your head is in anyway, right?
- [48:14]
Charles Axeldine
Yes, absolutely. And yeah, reading outside of work, I mean, I know people might be concerned about saying something like this, but I would say, hey, we usually spend a lot of time on our screens, so rather than spending mindlessly 30 minutes on a social network, why not reading for 30 minutes? And by the way, you don't have to always read technical books, right? Like, there are some really good non technical books that are also useful. One of the best ones I've read is Complication by Sergeant Atul Gabondey. It's an amazing book about how do you learn from mistakes and how to have a scientific process when it comes to learning from those errors. And there are so many other books like that. Another good example is like all of the aviation incident postmortem, you learn a lot of things about the human component of like an outage that are very applicable to software incidents.
- [49:10]
Gergely Orosz
Yeah. Especially because in software incident reviews, we try to keep them blameless. So we keep the human part absolutely out of it, which I guess it's good in some ways. Right. There's no finger pointing, but it can miss the very human nature of like, I mean, we're people, we make mistakes, it happens. And it's kind of to be expected as well.
- [49:31]
Charles Axeldine
Yeah, blameless is very important and the intent is right, but sometimes it's not well executed. And when you completely remove the human component, I think you're missing 80% of the incident. Potential mitigation.
- [49:43]
Gergely Orosz
Yeah, there was this incident where I remember it was a small outage at the middle of the night at 2am the engineer woke up and turned it into big outage because I mean, they were just tired. It was frigging 2am they thought they were doing the right thing, but because it was a blameless postmortem, we kept going about how the system could have prevented this. And I'm like, I'm sorry if you're like, what could have prevented this is like not having to make a decision or that NGO not deciding to try to mitigate it on the spot. And I mean, there are some things, but there was that context of like, look, when you're waking up in the middle of the night, either it could have been okay to ask for help or it could have been fine just to leave it and kind of come back into the morning. But again, because like this was one downside that I saw. And it doesn't always have happened like this, but in this case, that very important thing was missing. As a tired person, you're going to make mistakes.
- [50:39]
Charles Axeldine
Yeah. And being too blameless, or at least not involving the human component removes the opportunity for finding training needs. Right. And as an industry, compared to all of the other industry, we do so little training. So for instance, I was talking about like the feature flag. If somebody's never been trained on why it's a good idea to use feature flag, how to use them, what are the internal tools that you can use, what are some case studies, sure, they will not use it and it's a huge missed opportunity. So if you don't cover those aspects in a postmart end, you're missing those training opportunities. And yeah, we need to also be completely okay saying, yeah, mistakes happen and it's a way of life. And as my mom used to say, only those who don't do anything, don't break anything.
- [51:23]
Gergely Orosz
Yeah. Speaking about training, I had this really interesting experience with you. I'll see if you remember. So back when I started at Uber, we were in hypergrowth mode and we had a lot of projects going on and engineers needed to lead projects. We just didn't have the product managers didn't have the capacity to lead them. And so you decided like, okay, well since everyone's leading projects, we might as well get people some training on what does good project management look like. Because we're all software engineers, we're not the experts on this. You looked and you found an expert on this. You brought that person in for a two day training and at lunch break all hell broke loose. Engineers were like, what the hell is this? Like this guy is talking about some old school thing that doesn't work. The rules were so big, we canceled the training. And I still think of this of like, what went wrong or like, could have we even done training or were we just doing things that was just not trainable or would have we needed to do our internal training. Do you remember this?
- [52:26]
Charles Axeldine
Absolutely. It's funny, I absolutely remember this. It was a very humbling experience because I did choose the training company. The main learning for me is you have to build your own training. Only you know what your team needs. And the mistake I made was to to have somebody else come in and do a training because you spend a.
- [52:48]
Gergely Orosz
Bunch of time selecting who to bring in.
- [52:50]
Charles Axeldine
Yeah, no, no, I mean a big mistake for sure. Since then, very simple. I just build my own trainings. I know exactly what the team needs. And the other thing is, I keep on improving it. The true thing is, like, project management is not rocket science, right? Like, we could go into details, but it's essentially managing the trade off between time, cost, and quality. And once you've said that, now you need to apply it to your team, give case studies, have people engaged, and only you can build those trainings. Right.
- [53:18]
Gergely Orosz
And then that's what we actually did afterwards. I mean, I don't think it was intentional, but I started doing this. I started to put a document together for my team saying, okay, here's how we usually do it. Here's examples, here's like, pointers to internal sites. Here's how we try to keep it as lightweight as possible. Because I think that's the thing with a lot of these things. I wonder if this is why training is possible. As an engineer, you want to make.
- [53:39]
Podcast Narrator
The process as simple as possible.
- [53:41]
Gergely Orosz
You need to have some, you know, like, I don't know, checkpoints. Sometimes they're optional, but in the case of project management, for example, I think the only thing we had is, okay, you know, kind of let's do some planning up front, decide how the hell how you want to do it. It's up to you. Do you want to do a big meeting? Do you want to do whiteboarding? Doesn't matter. Put together a plan, make that a document, send that out for comments. That was like kind of a fixed thing. And then I think the only thing we mandated is have a kickoff meeting, because we realized that when we. We don't have a kickoff meeting, the projects don't go that well. And then I think the third thing we said is, like, every week, send the weekly summary to your team, everyone on your team, and a few stakeholders. And then there was a bunch of recommendations in between. You could do this, you could do that. But these were the three things. And we did it because we, over time, we figured that this works for us. It might not work for another team, might have added additional things. Again, I know some teams, they mandate, let's say, retrospectives and all of those things, but. And in the end, every team kind of came up with their own process. Some teams kind of forked it or copied this document. They added some things, they removed some things.
- [54:43]
Charles Axeldine
Yeah, you have a ton of good content on your blog about project management. Maybe one thing that most people don't realize is this weekly update thing, which is something I learned at my first job, an internship in South Africa. And my manager was sending this weekly update to almost the whole company. It does so many things. Right. So one, it forces you to be explicit about your goals, your success, your lowlights, what you can improve. Right. So that's great. And two, it's a good perception tool. It's incredible how just sending a weekly update drives a perception of movement. And it's critical. You have to manage the perception of your work. It works for a team. It also works for individual contributor. You've been tasked to do this or that project, very simple move, send a weekly slack message. This is what I'm planning to do this week. This is what I did last week. That's it. Pretty much like a stand up and maybe some key metrics. Right. In writing it works wonders.
- [55:43]
Gergely Orosz
Yeah. And I really liked how you suggested that we do. We did for every week we did like the status of, you know, like is it on track? Like when are we planning to have the next thing shipped? Or whatever that was highlights some good things. And then we also had low lights. We just tried to keep ourselves honest. And what I actually found is by adding lowlights and by being honest and of course with lowlights, when there's something at risk like mitigation, what are we doing about it? It just kind of forced conversations early. We didn't really have surprises when a project was late. We knew it well ahead and when we didn't, we then talked with the team or the engineers saying did you actually notice? And usually turned out that they kind of knew. They just didn't want to bother with it. But by having that trust again, it wasn't about like, you know you're going to get in trouble for being late. It's like, no, we just like to know so we can help, we can talk about it. We can talk about the trade offs. Because it's all trade offs. Right. Like with project management, you can always decide when a project is looking that it's late. You could either cut the scope, you could try to pull in more people, you could just extend the late. The timeline. Yeah, I mean that's. I think that's the three things you can really do.
- [56:49]
Charles Axeldine
Yeah, absolutely. And the low lights is an important section. I don't always include it, frankly. But one thing I always make sure to include is ETAs. A deadline is the ultimate source of inspiration. Right. Because the thing, if you look at the cost, which is essentially the engineering time, it's always a dynamic thing. How many engineers are working on the project is not, not always like super easy to measure the quality, which is the scope and what the project is aiming to ship. Is also very dynamic. Like you have macro optimization at the feature level that people might not have the date. Super objective.
- [57:24]
Gergely Orosz
Right.
- [57:24]
Charles Axeldine
We said we would ship on that day, we're going to be two days late and then the discussion about is it okay, should we cut scope, increase the number of engineers which does not always work.
- [57:35]
Gergely Orosz
Right.
- [57:35]
Charles Axeldine
Mythical man months. But then those dates and making sure they are in your weekly update. Ultimate source of inspiration when it comes to project management trade offs.
- [57:46]
Gergely Orosz
And I feel that getting better at project management, managing your own project that you're working on, whether it's a one person thing or a few people, it's a superpower as a software engineer because I think as software engineers we think that okay, our job is to write code, but actually our job is to solve problems and to ship solutions to problems that the business has. May that be new features or products and you become more experienced and frankly higher paid when you can take on more complex things. And there's only so far you can go with that project management. So I remember that at Uber there was some pushback. Some people said like I don't really want to do project management. And I was like well okay. So I guess two things can happen. Like one is someone else can do it and you'll just, you're not going to do it. Or like two, we could bring in a dedicated project manager. We could literally hire a person and they will be the project manager. Non technical. They're going to keep pestering you for updates. You know, you're probably going to hate them.
- [58:42]
Charles Axeldine
Yeah.
- [58:44]
Gergely Orosz
But a lot of places still work like this and some of it has to do with some people are just really resistant to. I don't want to do it. Great. Well, you might actually be worse off.
- [58:55]
Charles Axeldine
Yeah. I've never worked at a place where you have those non technical project manager like you. I've been. Usually you have this TPM role but it's most effective I think when you have like a project that impacts 20 or 30 teams, very useful. But then when it comes to like once again it's not rocket science. It's like a weekly updates, making sure the project is on track, making sure it's still going to be delivering the value that people are expecting. And I would even go as far as saying that project management is one thing but I would also recommend getting into product management. Right. Because the irony of the software engineers work is there's a lot of blame, a lot of things that can be blamed on the software engineer. Right. Like a feature is buggy hasn't been tested enough, there's an incident, a wrong architecture decision was made. And it's very objective how you measure. There is one thing that is relatively difficult to measure and that most companies I think don't do a great job at, which is making sure a product is doing the right thing and was the right strategy in the first place and is shipping the right value. So as a software engineer, if you don't do that, you risk wasting not days, but weeks, maybe months of time working on a feature that is actually not going to move the needle for the customer.
- [60:06]
Gergely Orosz
Yeah, or even years. I think we see products that even like larger companies bring out, which are just like big flops. And they put so much time and effort into this. I mean, we could even see it at Uber projects that were killed. I still remember to this day date, we had a team who were doing jump bikes and they were integrating it into the Uber app and they were changing some parts of our stack and they came in and they didn't add tests. Like they added a feature or something and they didn't add tests. And we kind of got into standoff with them because they were like, I mean, you know, like our quality bar, it needs to be test. In fact, I think they deleted tests even worse. And at first I was like, how can they call themselves software engineers? We're at the same company if they miss these quality things. And for us it was really important because we knew that we want to have stability, we have the real world impact and so on. After a while when I talk with them, I started to understand that they're coming from a different angle. They were not sure if their team would exist in two months and actually it didn't in like six months, it was kind of sold off and or move to a different entity.
- [61:17]
Podcast Narrator
But they were just in a different stage.
- [61:19]
Gergely Orosz
They were in survival. They were like, let's just get it to work. Let's test it if it's there. And this is what made me realize that there's just different levels of software engineering. When you're working on a mature product that's making a bunch of money, which was us, you have different goals than when this was a startup and MVP stage, they weren't sure if it would even work.
- [61:39]
Charles Axeldine
Yeah, the time cost quality management is very different depending on where your product is. So if you don't understand that, you're not going to be making the right decision. In particular, when it comes to technical debt, taking on technical debt is a perfectly valid thing to do when you don't know if your team is going to exist next month might be much trickier if you're building the architecture for the next few years. And I would say especially the case for internal projects. Especially because internal projects, you have an internal platform. Usually you don't have product manager. So in a way, you are also the product manager.
- [62:17]
Gergely Orosz
How would you suggest software engineers to learn to kind of make a balance on tech depth, on how much to focus or how not to focus or what is your mental model of deciding how important is tech depth or not? Because again, I've seen most engineers, I think, that care about the craft can frankly go overboard on this. You're trying to fix it, make it perfect, but honestly, it doesn't matter all that much. But on the other side as well, right, we have this kind of like, kind of hacker mentality who just gets things done. And yeah, it's a mess. It's spaghetti code. It's hard to maintain after.
- [62:49]
Charles Axeldine
Yeah, I would say two things. One, it's tech depth. If you know it's tech depth. If you don't know it's tech depth, it's just recklessness, right? It's moving fast and not knowing what you're doing. It's. I think we call it vibe coding those days, right? That's not tech debt. It's just recklessness. So you have to know, you have to care a lot about your craft to know where you're taking a shortcut. And two, I would say condone the technical debt, right? Like, hide it behind a clean interface. Use good patterns to make sure that then you can revisit that technical debt with minimal changes and keep track of it.
- [63:22]
Gergely Orosz
Evidently, one thing you've done a lot is hiring. You've hired so many people over time. Right now the job market is. It's kind of bouncing back a little bit, but it's still pretty tough out there as software engineers, as engineering managers, to get hired. What is your advice on a software engineer, let's say experienced software engineers, to get hired? How would you suggest that they prepare?
- [63:45]
Charles Axeldine
Yeah. So first, you're absolutely right. Like, the press and the media is full of articles about lowering, like, a difficult job market for engineers. You've shared an article yesterday, but it's not what I'm seeing. We are hiring, for instance. So I would say don't listen to those news. I mean, there are so many advices on the Internet about preparation and things like that. So it's difficult to say original things about it. But I would say definitely is to have a Structure and a method you come in. Most of the interviews you will have are technical interviews. Hands on. If you have a good method, if you have a good structure, the interviewer, even if you don't succeed necessarily at solving the problem at hand, will see that and will reward you for that. I think so. For instance for algorithm it's to start with a brute force solution and then to keep improving and optimizing. Right. For a system architecture interview, it's to start with the product, then go with the requirements, technical and non functional, functional and non functional and then start with an architecture etc. Focusing on the trade offs and things like that. So having a structure method, I would say definitely super useful. Then there is the topic of AI. Do we want to talk about AI and interviewing? Top of mind.
- [65:02]
Gergely Orosz
Yeah, yeah. But before we get to the interview, when you're hiring on your team, I guess it starts with either applications coming inside or you also have so called outbound sourcers who you talk with them and you tell them, here's the type of profiles I'm looking for. What could help you being discovered. I guess this will be your LinkedIn profile. And also for resumes, is there anything that again there's been so much advice. I even have a book about resumes, but is there some resumes that you think are better than others?
- [65:37]
Charles Axeldine
The main thing people will look for is what are the experiences. Which obviously is not a valid but it's still a price for people who are just out of college, for instance. But I would still recommend that they promote the projects that they've been doing.
- [65:52]
Gergely Orosz
And for instance, well and also I think this is important because when you're choosing your next job, if you have options, you could think about what will help my resume stronger. All things being equal or similarly equal. I had this early on in my career when I had the opportunity to join Skype. I actually it was the first job that I didn't take any pay rise. It was the exact same thing but I would have taken the pay cut to do so because I, I knew that that was the first company that I actually recognized. And later once I got there, suddenly messages started to come in because I was now at a company at the time. It was, it was well known, you know, nowadays it's discontinued but still, yeah.
- [66:30]
Charles Axeldine
You pick the right first expenses. You're absolutely right. Like you came up on our radar because of like the expenses that you had. So I would say go to what companies that are known for the quality of their engineering because that's what but that's going to be the main feature as far as like resume and LinkedIn, I mean, don't overthink it. Most people don't spend too much time on it. Don't choose AI. I would say don't use AI to generate your resume. I've spent the last few days reviewing 30 resume. 25 out of those 30 were AI generated. Some even add in the skills attended standups and there were a lot. So now I can tell like an AI generated resume, it has of a lot, lot of like those relative number improvements. So there's going to be reduce deploy time by 30% and no absolute number which reduced by 30%. I mean if you reduced it from two days, it's, it's, yeah, sure, it's not very impressive. 30%, right. So don't use AI. Keep it simple, keep it short, keep it creative as well. Right. Like we like sourcers and hiring managers and recruiters, they review hundreds of resume per day. So if you want to stand out, be original, be creative, don't use AI.
- [67:45]
Gergely Orosz
Yeah. And I think maybe like as you're looking, it could be worth looking at other people's LinkedIn profiles and just making notes of things that you think are kind of like cool, original and like you can use some of that as inspiration.
- [67:58]
Charles Axeldine
And this goes back to the choosing the right company to work for, which is what are the technical problems that are going to really move your career forward. So that's why also it's a good idea to prepare questions for your interviewer. And one of those questions should be what are the technical challenges that I'm going to be exposed to? Because then it's going to give you a good direction for your career and you're going to be able to check whether this company makes sense or not.
- [68:25]
Gergely Orosz
And then once you're on the interview and you're talking with, I mean the technical interviews, I think that's pretty straightforward. You just need to be good at it. But when you're talking with the hiring manager, someone like yourself, you know, like who are people who do pretty well on the interviews? I'm assuming don't ramble, for example.
- [68:41]
Charles Axeldine
Yeah, I would say don't ramble. And also short answers. Right. The dynamic of the interview, it's the same here, is like somebody's asking questions and driving the conversation. It is what it is. So if you use all the time, the interviewer is not going to be able to ask that question for the hiring manager. I mean, it depends if it's a cell call or if it's a behavioral.
- [69:03]
Gergely Orosz
Interview, by sell call we mean there's calls where you're not sure if this person is interested but you would like to convince them to apply. Typically for more high profile people, oftentimes industry known, maybe people who are at a company that is known to have really good benefits. For example, someone working at Meta right now, it's kind of hard to sell them because the stock is just going upwards. So their compensation might have gone up. So that's a sell call.
- [69:27]
Charles Axeldine
Exactly. And then you can prepare by asking smart questions. Right. So you research a company and you have a good list of questions to ask. I think the best candidate like prepares the interview. Even those conversations with the hiring manager. And what this says about the candidate as an interviewer, it says that if this candidate prepared the interview really well, that means that they're probably going to also prepare their work really well when they are at the company.
- [69:55]
Gergely Orosz
It's kind of surprising that we even have to talk about these things. But I talk with a recruiter at a publicly traded tech company. There's a few thousand people there. They're a well known company. I just don't want to say the name because I don't want to put them on the spot. But this recruiter said that they were recruiting for director and VP level candidates and the people who were coming in did not know the company's products. Like this company makes a bunch of different products and she said that 90% of people just didn't, didn't do the basic research of like. And we were, we were not talking about software engineers. So I think I'm going to just plus one that, that when you get to an interview, first of all it will be a red flag when you have been ignorant and you actually interesting enough, you will stand out by putting in a little bit more than the basic preparation which by the way could just be interesting to learn about different business, what they do think about what their business model could be. This goes back to, you can actually use this later when you're in an elite position, maybe one day you'll become a founder. All useful information.
- [70:54]
Charles Axeldine
Yeah. And you have to manage your career.
- [70:56]
Gergely Orosz
Right.
- [70:56]
Charles Axeldine
So if you don't know anything about the company you're interviewing for, you cannot know whether this company will actually empower your career or not. So this preparation is also in your interest.
- [71:08]
Gergely Orosz
Well, yeah. And also preparation is in your interest to see how the company is doing. For example, financially and business wise. It would be nice to know if there's a likelihood of the company doing cuts in the next six to 12 months, which might be hard to tell from the outside, but if you do zero preparation. On the podcast, I had Gianviklarna, who's at OpenAI, a software engineer she interviewed at 46 different companies and she researched all of them. And then she made this list of like, based on. She actually researched their products, she tried to use it, she looked for Reddit reviews for all of these to decide finally where she would go, because she wanted to go to a company where it has a good trajectory, you know, her stock could be worth something, et cetera. And I realized, well, yeah, I think more people should do that.
- [71:51]
Charles Axeldine
Yeah, if you're that careful about your career choices, I think that says something also about like your qualities as a software engineer. We were talking about like a previous colleague of us. Right. Like who has the same structured mindset when it comes to picking their next startup.
- [72:10]
Gergely Orosz
Right.
- [72:10]
Charles Axeldine
And that applies to everything. You have the same structure and method for picking your next career opportunity and also to pick your next technology to use in this or that architecture.
- [72:20]
Gergely Orosz
So with AI, how is AI from your perspective creeping into the hiring process? It's everywhere, right? You already told me that you can already see the interviews, sorry, the resumes that have been written or prompted by AI. But what else are you seeing?
- [72:38]
Charles Axeldine
Yeah, we see it a lot in interviews as well. So for instance, we explicitly asked not to use AI. It's actually quite interesting. I think we'll go back probably to on site interviews in the industry a lot more than we used to.
- [72:50]
Gergely Orosz
It seems unavoidable.
- [72:52]
Charles Axeldine
And by the way, this is not to say that AI is not a good tool when you're on the job. It's just a really bad tool when you're interviewing somebody and you're trying to get signal because you don't know if you're only testing their prompting abilities, which is a very important skill, but it's not the only skill. So that's one. So I would say use AI as a coach, as a trainer, but certainly not as a cheating partner. And as a trainer, it's a really powerful move. Right. Like for instance, you can prepare those questions about the company and ask whether those questions make sense. And you can also prepare, you can prepare an architecture interview. There is so much you can do to prepare and train yourself, but I would not use it during an interview.
- [73:33]
Gergely Orosz
So you've worked at a couple of fast paced startups, Uber, you later worked at other startups, you're now at cloud kitchens. What have you seen People do to be successful at these startups. I've seen when I hired people at Uber, a lot of them hit the ground running, but some of them struggled because it was just a big pace change. And these are all fast paced startups and you know, hypergrowth is probably over, but fast paced startups are here to stay. In fact, what I'm seeing with startups of today, they're even faster, they shift faster, they get to product market fit faster. What have you seen engineers do who do well in this environment?
- [74:09]
Charles Axeldine
I would say first, personal productivity helps a lot. So investing in this early in your career so that then you can hit the ground running really rapidly and you stay structured and methodical when you're on board on a new company.
- [74:22]
Gergely Orosz
And sorry, by saying investing in personal productivity. So figuring out how to what works for you, like what kind of methods may that be to do lists or pomodoro or focus time or whatever.
- [74:33]
Charles Axeldine
Exactly. Yeah, yeah. So how do you keep track of your commitments? How do you keep track of your reading?
- [74:39]
Gergely Orosz
What worked for you or what have you kind of what phases have you gone through?
- [74:42]
Charles Axeldine
In this sense, I've mostly used a method called getting things done, which is not rocket science, fairly simple. And then I've used the same tool which is called Things for the past 15 years, I think a plain todo file and then I also use a GitHub repository for my personal notes that's called PKM Personal Knowledge Management. That helps a lot as well. And I try to memorize stuff as well with flashcards. Anki. Those are things that I've used consistently and it pays dividends still to this day.
- [75:17]
Gergely Orosz
What kind of things do you memorize?
- [75:19]
Charles Axeldine
So for instance, I memorized the whole Python standard library and not everything but a lot of it. And it did help a lot in the beginning. And then you memorize like architecture patterns, data science methodologies.
- [75:35]
Gergely Orosz
Yeah, love it.
- [75:36]
Charles Axeldine
Standard library of your current language or like some issues you have bash commands that you use consistently. Yeah, so I would say personal productivity does help. Reading fast and being able to synthesize and summarize like document, really I guess.
- [75:53]
Gergely Orosz
The best way to do it is to practice. Right. So like read. Read.
- [75:58]
Charles Axeldine
Exactly. Definitely. Prioritizing meeting with people and understanding their context because there is only so much you can find in the internal documents and in the call. Right. So understanding the history of the technical decision helps a lot when it comes to maintaining that platform later on.
- [76:14]
Gergely Orosz
So I'm not sure if you recommended this or not, but at Uber, one thing I started to do after a while is was meet when I became an engineering manager, meet other engineering managers on the team or when I went to, for example, San Francisco, we were based in Amsterdam. Spend that time to meet with them in person. And then starting with like I did a short summary of short presentation of our team, just a few slides to say, here's what we do. It was meant to be really short, but it was just a conversation starter. And then I asked like, hey, what do you do? How did you get into Uber? What are your plans after? And then as we warmed up, what are your plans after Uber? Like, where do you actually want to go? And those things have become really useful. And I realized as an engineer I probably should have done more. And I saw some of the best engineers, they actually just met in person with another tech lead or another engineer on the team or grabbed lunch with them and it makes all the difference.
- [77:03]
Charles Axeldine
Yeah, I think AB talked a lot about it.
- [77:05]
Gergely Orosz
I think it makes sense she might have been the instigator.
- [77:10]
Charles Axeldine
It makes a lot of sense. Right. You're going to work with people, you're not going to only interact with code. So getting to know them on a personal level is critical if you want to work well with them.
- [77:19]
Gergely Orosz
And they'll just be nicer to you.
- [77:21]
Charles Axeldine
And it's nicer. Yeah.
- [77:22]
Gergely Orosz
And you'll be nicer to them.
- [77:23]
Charles Axeldine
Yeah, yeah. So that's why, coming back to this like humor and self deprecation, you want to work with colleagues who are really good at what they do, that's for sure. But you also want to have fun, right. Working with them. Like it shouldn't be like an annoying thing or frustrating thing. So the best way to do that is to get to learn because I'm convinced that everybody has an interesting story and stuff you can learn from them.
- [77:50]
Gergely Orosz
Yeah. And I guess one thing, in a fast grow company, conflicts are a bit more common just because people try to get things done. There's a lot of email or slack messages and sometimes they can come across as aggressive or passive aggressive. And I still remember to this date where this was again back at Uber, we were emailing a lot back and forth with the San Francisco platform team and there was this guy who was just an absolute asshole at times, just like in the response. And then once he was in Amsterdam and he was such a nice guy, but once we got to know him and someone asked, why do you write this? Like, oh, I just write really fast. It was really late at night. Turns out he was writing at 3am and he wasn't caring too much. And every now and then I get back to the point of, well, I guess software is oftentimes more about people or at some point people start to become really important, especially inside a company.
- [78:42]
Charles Axeldine
Yeah. That's why my number one advice when a code review is not going well or like an RFC review is getting into tons of comments is just meet. Right. Like meet, meet face to face. You're going to hash this through in less time than it's going to take you to have all those exchanges and frustration and everybody's second reading the thing. I mean let's be honest. Also a lot of people in startups are like English as a second language. So we don't necessarily understand all of the nuances of the language, contrary to cod. Right.
- [79:13]
Gergely Orosz
And also some people cannot express their own nuances or they have a hard time, especially in writing.
- [79:19]
Charles Axeldine
Yeah, no, yeah, absolutely.
- [79:20]
Gergely Orosz
What are some non obvious recommendations to thrive inside a high growth startup?
- [79:25]
Charles Axeldine
Yeah, non, obviously. So I talked about memorization. Yeah. Using flashcards to memorize stuff. Very powerful. And second, I'm not sure if it's super non obvious but extreme ownership. So that means you feel like the team is yours, the project is fully yours and really going out of your way to understand all of the dependencies and overall context it fits in is really critical to be super successful. I talked about the best engineers don't stop at team boundaries, they will keep going. Our industry, our work is mostly about building abstractions on top of abstractions on top of other abstractions. In reality there's always leaky abstraction details, implementation details. Right. So the more you understand about the stack you rest upon, I think the more effective you're going to be as an engineer. It also applies to your tools, it applies to your programming language, it applies to everything.
- [80:20]
Gergely Orosz
Yeah. And then one thing you used to say a lot is under promise and over deliver, which I guess is easier said than done.
- [80:27]
Charles Axeldine
Yeah. This is where also the weekly update helps is like you set your timeline and if you find a creative way to reach a timeline faster, it's a.
- [80:36]
Gergely Orosz
Pretty powerful move when it comes at working at high scale, fast paced startups. A lot of people have imposter syndrome. Did you have this and did you see people have it and how did you work around it? How did you the people conquer this, who you work with?
- [80:56]
Charles Axeldine
Yeah, you always have this. I think to this day it's a good thing to have it. Actually we often talk about the imposter syndrome as a bad thing, but I think actually it's the engine that drives your curiosity and is what's going to move you toward essentially improving yourself and having this continuous improvement mindset. What's really interesting also in the world of startup mindset and culture and things like that is when you look at it, I've been really interested in ancient philosophy and stoicism and when you read those like ancient philosopher reading, it's still very applicable to us today. Right. Which is don't over focus on your emotion but just get at it. Right. Just try to work on those aspects so that you become a better engineer. I want to do just one quote, which is Dan Heller, which we both know is a really good way to phrase this thing. I think imposter syndrome is underrated. A lot of talk goes into overcoming imposter syndrome. I say embrace self skepticism and doubt yourself every day. What else to add, right? Like it's a good drive, it's a good engine, even Tuan City of Uber acknowledged it. And I think it takes a lot of humility to say that you feel like you have imposter syndrome and it's a good thing.
- [82:14]
Gergely Orosz
And I guess you typically have imposter syndrome when you feel people are smarter around you, where you feel you might not be fully up to speed with wherever you are, which is typically at a high growth startup like it's going to be a rocket ship. Right. So I guess it's kind of natural to feel that. Right. In fact if you didn't feel that like you might ask, are you at the right place? Is this really truly a high growth startup which has ambitions beyond your own ambition?
- [82:37]
Charles Axeldine
Yeah, you want to be impressed by your interviewers for instance. That's a good way to know whether you're going to be at the right company.
- [82:44]
Gergely Orosz
Next, what are some non obvious reading recommendations you might have for engineers who want to get better?
- [82:50]
Charles Axeldine
Yeah, so I would say read the fundamentals, not necessarily only the most recent technical books. Right. So for instance, one of the books that I non obvious book I would say is like the Linux programming interface which goes into what is the API that is exposed to by the kernel. It's fascinating because not only does it explain the kernel and it's super useful.
- [83:12]
Gergely Orosz
Because most of our stack is built on top of Linux.
- [83:15]
Charles Axeldine
Exactly. So that's one and two. It also goes into the historical technical decision. There are some really interesting algorithms as well, like why did the kernel choose a red black tree for instance instead of hash maps for certain data structures. So it's really interesting, really fascinating. So fundamentals, fundamentals about Your programming language, adjacent fields. So I mentioned complications by Atul Gavande. Fascinating read. And I would say fiction as well. Read a lot of fiction first, because, I mean, it's not all about, like, work, and also because it trains you to be a better writer. Reading good fiction is a great way to improve your English, improve your writing.
- [83:54]
Gergely Orosz
Skills as well, and I guess your reading skills. You mentioned how it is helpful to be able to read fast and digest information fast. And I guess as I go into the gym and working out, you're now working your mind.
- [84:08]
Charles Axeldine
Yeah. I've also been reading a lot more philosophy lately. I mean, evidently software engineering is a very scientific endeavor, but there's a lot to be said also about the fact that we handle concepts. So it's a very conceptual work that is very similar to philosophy. And so, for instance, I listened to your interview with Dr. Osterhout about the philosophy of software design. Right. And he mentioned that at the core of software design, you have this decomposition of problem. Well, it's the core of human thought. It's the core of reason. Right. Like, reason is about distinguishing between different concepts. And I mean, it's not an irony that there is philosophy in the title of this episode, because it is what it is. Right. Like philosophy is exactly that. It's distinguishing between different problems.
- [84:57]
Gergely Orosz
That.
- [84:57]
Charles Axeldine
That's really what classical philosophy is about.
- [84:59]
Gergely Orosz
Yeah. In fact, when it comes to, for example, designing a new system, you kind of need to go down a path. Like, you can argue if it's a philosophy or not, if you're doing microservices or monoliths, but the argument can turn into kind of philosophical against. There is no one right or wrong. And I guess at some point you need to go with one of them. It's fascinating how there are some parallels. Yeah.
- [85:24]
Charles Axeldine
When you're writing an rfc, you're making a case. So then there is a big component of logic. How sound are your arguments? So, for instance, at Cloud Kitchen, in the competencies, we have a big section about truth. And we have one competency which I like pretty much, which is called qed. Right.
- [85:41]
Gergely Orosz
Qed.
- [85:42]
Charles Axeldine
A quad era demonstrantum. What needed to be demonstrated. And it focuses on how well do you make your case? Do you eliminate bad arguments? Because when you read an RFC and you have one good argument in the middle of 15 bad ones, you feel like, yeah, I'm wasting my time. As a reader, it helps you uncover that, and as a writer, it helps you realize that. So that you focus on this good argument that you have and you make it really strong and it really helps your career.
- [86:11]
Gergely Orosz
I think it almost feels like if you're a good debater and you're also good at laying out your arguments in a logical way, it helps you be a good software engineer. Because we need this. Right. It's a logical field where the best people like, obviously there's a human part, but you need to build up your arguments in a way that's understandable, easy to read if it's in a document, which a lot of it is.
- [86:33]
Charles Axeldine
Yeah. There's two virtues that are really important and I think under focused in the startup world is courage, like the strength to go against difficulties and maybe opposition and humility. Because also when you're focused on logic and truth, you can also say, hey, I was wrong, I mean, sorry. Or like you make a case, somebody comes back with a really strong rejoinder and you say, well, yeah, you know what, you're right. This is not the right architecture and it's a great way to grow. Right?
- [87:02]
Gergely Orosz
Yeah. And the best engineers always share that. No matter how senior or experienced they are or how fancy title they had, they had an open mind and they would change their and they would go like, okay, yeah, let's build this idea in or let's go with your idea. Zero humility. A lot of the advice that comes out of people who worked at Uber at a hypergo time or today it'll be at OpenAI. We could argue a little bit. Is that survival bias or not?
- [87:28]
Charles Axeldine
It is. I think it is. It's definitely survival bias. You listen to us, for instance, and we're going to talk about all of the things we need to fix some of the chaos. But in a way you could argue that the chaos is probably why the startup was successful in the first place. Right. The speed up that you get when you are focused on shipping definitely has some trade off in terms of quality and maintainability and reliability. But I would submit that in most cases it's probably not what you're looking for. You are looking forward to first building a product, building a successful business and the rest comes after. And I think the industry has a tendency to over focus on what comes after, the quality, the investment in the internal platforms. Well, actually, yeah, it's fine in the beginning if it's not standardized, if it's not that structured, if what you get in exchange is speed.
- [88:26]
Gergely Orosz
I think Uber had this thing which we just got rid of just as I arrived. It was called the ping. Every five seconds or so, the app would send a ping to the server, it would ask me, send me the data, and the server would return a blob of information, the waiting times, the EDAs for the different categories and a bunch of other things. And the app would display that data. So this was a pull mechanism. And initially it started off with a few pieces of data. And they did it because the mobile developers were tired of waiting on the backend developers and they could just add things into this ping package. And it became a pretty large package and it was just causing a lot of Data. And in 2015, Uber was still working like this. It was now scaled to millions of people and it was just really, really inefficient. But then eventually, I mean, it kind of worked and a refactor came and then Uber changed it to be a poll where the. So it was pushed. So the server was now pushing and it was far more efficient, faster, shorter delay, etc. But in the end I was like, it worked.
- [89:29]
Charles Axeldine
There are so many examples like that. We were talking about internal tools and automation in the beginning, right? I think when it comes to automation, a lot of automation projects over focus on quality and constraints, when actually people want speed and being in control. There is this fascinating article about malleable software, software that leaves a user in charge. Best example being spreadsheets. So many internal tools start with a good Google spreadsheet because there is so much you can do, right? You don't need to ping an engineer to add a column to your table, you can just modify it. And what's great with spreadsheets is it's the purest. What you see is what you get, right? Like it's everything in front of you. It supports mass change. I would submit that a lot of startups have an incomplete buggy implementation of Google spreadsheets somewhere in their tools because it's so effective. One interesting story from Ubertime, the users table and the trips table had two interesting columns. One was called user and trip tags. So user tags, trip tags. Another one which was a freeform list of tags. And another one was user attributes and trip attributes, which was a key value stored as a JSON in this column. It was insane when you think about it, right? And it was massively overused. A lot of team did not want to add columns to the trips and users table. So what they did instead is just stick stuff in those columns.
- [91:00]
Gergely Orosz
Free text, pretty much blown.
- [91:01]
Charles Axeldine
And then it started exploding, I think roughly after I left, which is fair. But so many products were bootstrapped thanks to that. It's absolutely amazing. I think so that's why those really shitty internal tools approach, they do work because you get speed out of it and you might validate very quickly that actually you don't need this feature or product in the first place and you're going to save a ton of engineering time, which is usually the limiting factor.
- [91:29]
Gergely Orosz
So is it fair to say that what you're saying is if you're at a fast growth place, instead of reading what other fast growth companies did, their blogs, their best practices, maybe you're better off just looking at your most pressing problems, solve that and then do the next thing. Or also, or as with anything, there's a balance here. Or just be more like be skeptical whether the company like Uber or now we're going to hear about OpenAI or all these very successful companies when they say why they were successful. Maybe that's not the full story.
- [91:58]
Charles Axeldine
Yeah, I would say because usually the limiting factor is the engineering time. I would say optimize for leaving the user in control so that they can change things on their own without having to involve engineers too much. Put the guardrails in place so that it does not catastrophically break your system, but leave the user in control because you will get so much better. Product adoption by user, I mean usually internal operations team, internal business team, your product manager, lean on this direction rather than having a super constrained product that takes a lot more time to ship and that might not even solve the problem perfectly because you are missing this or that ops context, particularly the case when you build a product for the physical world.
- [92:44]
Gergely Orosz
So one of the big changes we're having obviously is AI. AI coding tools, AI use everywhere. What do you see? How is AI already changing software engineering, especially for the work that you're doing at cloud kitchens? And what are your thoughts on how it could change as it just becomes better?
- [92:59]
Charles Axeldine
Yeah, I mean so much has been written. Every week there's a new article, new analysis, new prediction. We did our own study, we found it was moderately useful. It is useful. So the thing I can add is for myself, I think it's particularly useful when you have a well defined problem that you're trying to solve. Navigating the code. As a coach and as a trainer, as a manager, I use it a lot to review my document. As a matter of principle. I never copy paste text prose that was written by an AI. I want to make sure I stay in control of what I write because yeah, writing is thinking. Right. It's the same process. So I don't want to outsource my thinking to an AI. I would be really, really sad if that was the case.
- [93:50]
Gergely Orosz
And when you're saying with engineering you found moderately positive impact. Is this to do with kind of using autocomplete, using it to generate some parts of the code? Did you go further in terms of let's say code reviews or trying to autom workflows?
- [94:05]
Charles Axeldine
Yeah, so we use it for all of those use cases. I mean, obviously I don't write as much code as I would like to and as I used to so well, but now you could actually. It's a great point. I did a lot more coding lately because with Agent in particular, it enables you to multitask. You can. You have 10 minutes before a meeting, you issue a specific prompt, you have your meeting, you come back and then you review the code. Yeah, I would say coding wise, refactor is really good at that. Refactor, well specified APIs and interfaces. Anything that is more complex, I feel like it's still failing. I'm sure it's going to get better, but I think the design component will always be a human being thing. The other thing, every single time we have one of those revolutions and AI might be the biggest yet, don't get me wrong, but every time we have those revolutions, the press and everyone speaks about replacing engineers, we're still there. You remember when machine learning really picked up, people were saying, yeah, so we won't need software engineers anymore. The truth is actually it's a data scientist that was really a job role that was under a lot of challenges because of machine learning and ready made models. But yeah, I don't think.
- [95:26]
Gergely Orosz
And for example, your team is still hiring, right?
- [95:29]
Charles Axeldine
Yeah, we're still hiring. I think it's going to enable engineers to do more, to focus on more interesting tasks. We talked about migration. I think migration is potentially a great use case.
- [95:40]
Gergely Orosz
We hate doing it and it needs to be done.
- [95:43]
Charles Axeldine
Yeah. So migration, great use case. So for instance, at cloud kitchens we had a migration and the teams that was running the internal platform put together a prompt that you could copy paste to do the migration and still review the cadence. Because there's always things that, that go wrong.
- [96:00]
Gergely Orosz
What about security or an increase of. Would we have increase of attack surfaces by using AI?
- [96:08]
Charles Axeldine
Yes, I think that's the most obvious thing as well. Security engineer is going to be a great job role. There's going to be so many things lately the main source of a big source of vulnerability was the supply chain attacks. When you take control of one of the dependencies of a project by Typo squatting or by just taking control of the repo for instance and then this way you can get remote code execution on remote virtually thousands of repository well agents are that at scale so it's going to be an incredible time for security engineer. So far so good. There haven't been so many problems but we're going to have so many.
- [96:47]
Gergely Orosz
It's absolutely obvious we need to prepare for that. What about engineering management? I mean you mentioned how you use it to, to help with some of your documents but do you think it changes the engineering manager's role? It help it makes some things easier. Does it make some things harder?
- [97:01]
Charles Axeldine
Yeah, I use it as a coach. So for instance where I would have potentially pinged my manager for advice and maybe more basic stuff now I get a first review from an AI. It's hit or miss. Like sometimes it's extremely valid feedback, sometimes it's totally useless and it, it's a good first pass I think But I would say never copy paste text from AI. I think we're going to lose skills if we do that.
- [97:30]
Gergely Orosz
Well one thing I think more people are going to use it. I talk with engineers. One thing, a few things engineers don't like to do that much. Meetings, migrations, performance reviews.
- [97:39]
Charles Axeldine
Performance reviews. So if they listen to this I have reached out to some of my skip level engineers and I told them hey, don't use AI to generate the.
- [97:52]
Gergely Orosz
Feedback for your don't only use AI.
- [97:54]
Charles Axeldine
At least yeah, he's gonna find it funny but there's even one engineer who took some of my Slack messages and then took our competencies and asked the AI to generate my feedback review.
- [98:09]
Gergely Orosz
It's unavoidable of happening but as you say, there is a point of view of writing as thinking and there's a reason companies try to force a little time to reflect on, you know, your relationship with this person or what you think of yourself, et cetera.
- [98:22]
Charles Axeldine
There's also one thing which is nobody wants to read AI generated text. The moment I tell you that this text was generated by AI, you lose interest immediately. So I would much rather what I tell people who might be tempted to do that is give me the prompt. If you don't have time to write good English and stuff, which by the way is not that important. Tell me what you would have asked the AI to generate and that's what I'm going to read because this is where the data there is this irony, right? Like it starts with a very small prompt, it's bloated into a massive text by an AI and then somebody else will use an AI to summarize the content. Why don't we just exchange a prompt directly? We're not going to lose any information that way.
- [99:04]
Gergely Orosz
I feel it's an very interesting contradiction with AI, specifically with text generation. I wonder if we're going to see more. We might see some other things in software engine. Slowly repeat as we learn. Right. Because I feel this is. Yeah, you just want the original information, the prompt.
- [99:23]
Charles Axeldine
Yes.
- [99:23]
Gergely Orosz
We'll see if software might or might not be the thing. Because people don't really care about the software. Users don't care what exactly code runs under the hood. So maybe it'll be different until there is a problem.
- [99:34]
Charles Axeldine
Right?
- [99:35]
Gergely Orosz
Until there's problems.
- [99:36]
Charles Axeldine
That's why the other thing is. I think you talk a lot about that on your blog as well. The importance of code review. The first time I used AI to generate code, I did not realize that it would be my code. So I did not re read it, proofread it that carefully. And then I put the PR up and I got a lot of feedback and I was like, yeah, I agree with all the feedback. This is really bad. And what I did after that is make sure that you read the code that was generated so that it becomes yours. Because at the end of the day, if something breaks, you're going to have to do this anyway. And that's probably the most interesting aspect, which is reading code that you did not write and making it your own is much higher cognitive load than writing it in the first place. And as a consequence, that's why you are still going to need engineers. Because at the end of the day, the moral agent, the person making the decision and putting their stamp, is still a human being.
- [100:31]
Gergely Orosz
I wonder if an interesting second order effect of AI, it's very good at generating a lot of text from a prompt. You know, it can generate like pages of text. It's also very good at generating a lot of code based on a prompt. You give it a prompt, it generates the code. As a result, it's pretty obvious we're going to see a lot more code generated. If you think about, you know, the professional engineering teams that you worked on at Uber, at cloud kitchens, at the other startups, we spend a lot of time as an engineers to not write verbose code. Right. Like on code reviews, we will push it down. We don't want to copy, for example, functionality that exists elsewhere. We would rather use the abstractions. What do you think the impact might be if we just continue the thought experience that we will have a lot more code generated, a lot more wordy code than need be. Where will this lead?
- [101:15]
Charles Axeldine
I think those startup might fail because.
- [101:19]
Gergely Orosz
Or slow down. Right?
- [101:20]
Charles Axeldine
Yeah. Because if they're only doing wipe coding with very little code review, it's. It's going to become unmaintainable even for an AI. So I would say you have to stay in control. You have to have strong code review practices in place. Yeah. One thing I often see when I ask AI to generate, it's trained on stack overflow content. It's trained on not necessarily the highest, best quality code.
- [101:42]
Gergely Orosz
It's whatever's on GitHub or whatever's open source. I mean some open source is high quality.
- [101:46]
Charles Axeldine
High quality. Yeah.
- [101:47]
Gergely Orosz
But a lot of stuff that's open there and can be used for training is not.
- [101:51]
Charles Axeldine
Yeah. So for instance, you will see that it's reinventing a feature that you know already exist in the standard library or in this or that library. Very often a failure mode. I think so. I think this is one of the. Yeah. One of the risk is you end up with so much code that you actually default on the code. Like being so overwhelming that when something bad happen, you cannot debug it yourself. You have to involve other APIs that will fail. Yeah.
- [102:19]
Gergely Orosz
We might relearn some lessons that we've already learned as an industry a few decades ago.
- [102:25]
Charles Axeldine
Yes.
- [102:25]
Gergely Orosz
And as closing, what is a programming language that you like most and which one would you like to learn?
- [102:30]
Charles Axeldine
Yeah, difficult question because I love programming languages. I love this quote from Jean Stubbort, the creator of C. There's only two types of programming languages, those we complain about and those nobody uses. So in the first section I would say Python. Why? I mean, it's so effective a language, so versatile. A really great. By the way, interview choice. If you have a coding interview, it's a really smart move because it's so effective. Right. You can get so much done. It's a professional language nowadays. Might I remember. Yeah. Remind people that most of Uber was built on top of Python in the beginning. So it's a really, really powerful language. TypeScript is pretty good also in that destination. I usually prefer dynamic languages because they leave me much more in control. I love statically typed languages as well. And then learn. I learned Clojure. I never. So that's the second, I guess, category. I never got the opportunity to use it that much. I think it's a beautiful language. Lisp dialects are really cool. Love functional programming. But sadly I haven't had the opportunity. Rust is pretty good too. I learned it as well in that section.
- [103:44]
Gergely Orosz
Yeah, no great Charles. This was a really fun conversation. I'm glad we had it.
- [103:48]
Charles Axeldine
Yeah, same was a long time overdue.
- [103:51]
Podcast Narrator
I hope you enjoyed this candid conversation with Charles. One of the things I like most about him is how he keeps trying out new stuff and is an absolute productivity geek. I hope this came across a conversation of as well. For more tips on how to lead projects as a software engineer, check out the deep dives into Pragmatic Engineer that I rolled in the past, where plenty of lessons come from working with Charles.
- [104:10]
Gergely Orosz
They're linked in the show notes below.
- [104:11]
Podcast Narrator
If you've enjoyed this podcast, please do.
- [104:13]
Gergely Orosz
Subscribe on your favorites podcast platform and on YouTube.
- [104:16]
Podcast Narrator
A special thank you. If you also leave a rating for the show. Thanks and see you in the next one.