← The Pragmatic Engineer
Loading summary
Transcript235 lines
- [00:00]
Host
Kelsey Hightower is known as one of the most influential voices in the Kubernetes community, but you wouldn't guess from how his career started. At 19, he dropped out of college to be a DSL modem installer, became a self taught developer, and still went on to become a distinguished engineer at Google at age 43. He then retired at the very top of the industry. Today we cover Kelsey's unconventional path into tech and how he kept creating new opportunities for himself, often unknowingly. The inside story of the Container Wars Puppet, Docker, Terraform, Coreos, and how Kubernetes eventually won going from a Google IC to executive level and how he rejected a Microsoft offer from Satya Nadella himself and still doubled his compensation. His grounded and pragmatic advice for software engineers worried about being commoditized by AI and so much more. If you're an engineer and thinking about your long term career trajectory, whether that's getting into a staff plus level, going independent, or even quietly planning to leave the industry, this episode is for you. This episode is longer than a normal episode, frankly, because I was so glued to my chair, mostly listening to Kelsey's stories and thinking this episode was presented by Antithesic Verify your system's correctness without human review or traditional integration tests, and avoid bugs or outages. Before we start, I'd like to mention our presenting sponsor, Antithesis, and maybe offer a little history lesson. Over the last two decades, software development has gone through a mindset shift from an imperative approach to declarative. 1. Infrastructure is a perfect example. Think about how tools like Puppet and Ansible allow declaring how individual servers should be configured. Then came Terraform, the ability to declare the desired end state of your whole infrastructure servers, networks, databases, and their relationships. And then with Kubernetes, we stopped scripting container lifecycles. Instead, we write manifests that say things like I want three replicas of this application exposed behind a surface with this much CPU and memory. Once we didn't have to specify every little detail of our infrastructure anymore, deploying software became much faster. But then the bottleneck became. How quickly we could test and verify the code to be deployed to testing remained imperative. We had to write tests for every little detail. And now with LLMs we're on the verge of a declarative shift in the way code is written as well. Just tell the model what you want and let it figure out the details and it's going to make the verification bottleneck a million times worse. Antithesis is a declarative testing tool that can keep up with your AI coding agents. You state the properties you want your software to have and Antistesis figures out
- [02:24]
Interviewer
how to check them for you.
- [02:25]
Host
Verify your code as fast as agents can write it and ship with ground and confidence. Head to insistences.com Pragmatic Kelsey, welcome to the podcast.
- [02:34]
Interviewer
It's so nice to see you in person.
- [02:35]
Kelsey Hightower
Yeah, I'm actually happy to be here, mainly because I kind of look at your stuff over the years. So it's an honor to be here in Amsterdam as well.
- [02:41]
Interviewer
How did you make your first dollar at a job?
- [02:46]
Kelsey Hightower
Oh, my first dollar at a job. McDonald's. That counts. So in high school, you get the job that's closest to you. So it was in walking distance of my house. As soon as I turned legal, age 14, get a work permit and I went there and it was one of those jobs where, you know, you go, you fill out the application the same day, you typically get your information or you're gonna get hired the same day. When can you start? I'm like, right now. They go get a shirt for you in the back. And what size do you wear? Men's large. And the one thing I liked about that job is you're dealing with real people that are in a hurry. I guess. One bad part about the job, you know, a lot of people don't respect people who have that job, so they kind of look at you as just like this intermediary thing between them and what they want. But there's so many things that go into a restaurant like that is very efficient. You know, people have expectations. I learned how to run the whole store. So by the time I turned 15, I was a assistant manager. So nights and weekends, you know, other managers would leave, they would give this 15 year old the keys. And I knew how to do everything there, including close out the store. Right. So you had to count all the money. You have to fax this huge spreadsheet to corporate every night. And then my mom would pick me up. And so it was really good learning how to really be responsible, even for adults at that age. So that's how I got my first dollar.
- [04:01]
Interviewer
How did you get into tech? How did you get into programming?
- [04:04]
Kelsey Hightower
In high school, since I moved from California to Atlanta. So. Right. You're going from one side of the country to another side of the country. And I missed maybe three to six months of school. And in order to graduate on time, I had to take some extra classes. And so as someone who played sports, ran track, played football, played basketball, and it's like, you know, there's this computer programming, you know, computer club, technology student association. There was a class component, and then there was after school component. And I was like, I don't know, man. This computer stuff, that's for the. You know. You know, I'm trying to be a cool kid, but the one thing I did, I really enjoyed it, right? So I had a liken to AutoCAD. I even competed at the state level in AutoCAD. So we drove down and you compete. They give you a task, and you sit in front of the computer. It was my first year doing it, but I really liked the idea of, like, taking a specification, designing it, and I probably would have gotten first place if I would have got the product to work, because you also have to print it out so that the judges can review your work. So I got second place, even though I didn't AutoCAD.
- [05:05]
Interviewer
This is the 3D modeling.
- [05:06]
Kelsey Hightower
Yeah, AutoCAD. You know, just AutoCAD. Yeah. So part of the curriculum was, you know, you build bridges. We did this thing with chapter team where, you know, you have a coat of arms and you're kind of doing like a debate. Do you kind of learn all of these things related to business? But CAD was one of the things I liked most. Also in that class, one of the classmates taught me ti basic.
- [05:26]
Interviewer
Is that a version of basic?
- [05:27]
Kelsey Hightower
Well, so ti basic. So you know the graphing calculator, it's like a ti86. Oh, yeah, you can program it. Oh, right. So in class they were like, hey, you know, it's not just a graphing calculator. You can actually program it. And I was like, what's that? And it's like, look, we can. You know, every episode you would create the SNCC game, right? So it's basically get a magazine copy and paste the code, and then you run it. And now you're playing Snake based on the code you wrote. And so you would toy around with this concept. So that was probably the first introduction to programming, was literally programming my. My ti86 calculator.
- [05:59]
Interviewer
And after high school, did you go to college or you considered college?
- [06:03]
Kelsey Hightower
Right, I considered college because in Georgia at the time and still today, there's a thing we call the Hope program. So if you have a B average or above, you can go to any public school for free. And public schools in Georgia include Georgia Tech, Georgia State. These are pretty good universities. Yeah, they're really. And so I decided to go to one that was near me. The first two weeks, I was like, this is Too slow. This is not, this is not the pace that I want to move at. And also remember it's 1999 when I'm graduating. And so when you turn on the tv, people are standing in line for the next version of Windows. There's a lot of euphoria. AOL is starting to phase out and we're starting to touch on high speed Internet. And, and I was like, yo, look at, look at the pace this is moving. But also you're hearing the narratives. Bill Gates drops out of college. These people are not necessarily glorifying the degree anymore. It's all about the skill. Now, unfortunately for me, I didn't know anyone that was a programmer. I didn't know anyone that was like a system administrator because at that time all the systems were like Sun Microsystems or IBM Mainframe. Those are still the things that are in the enterprise. So when I look at the job openings, I'm seeing a bunch of skills that I don't even know how to acquire. And so instead of going to college, you know, I'm still doing fast food, delivering pizzas at this time at Pizza Hut. And I remember going to a bookstore and they had the A certification guide. And I looked in some of the job postings, said, hey, you need to be a certified, take this support role or whatever it was. And I was like, you know what? That doesn't require college. The book is only $35. And I remember buying the book and it has an official certification process. It looks like it was part of the job market. And so I remember buying that book and reading it cover to cover over and over again. And you're learning all the fundamentals, right? You're learning about, you know, how motherboards and how memory and all these things work. And there's an OS component and there's a little practice exam in the back. And so for someone like me, having that fast feedback loop of like you put the CD in, you take the exam, and even though it was multiple choice, you kind of felt like if I got anything wrong, I would just go back to the book and make sure that I understood what was written there. And then you would go take the test again. And it had a little randomization to it, so you couldn't just rely on absolute remembering everything. And I remember going to the facility to take the test and you know, you're in that little room and they want to make sure you don't cheat. So there's a camera pointed at you and you're just going through. And so the nice Thing about those tests, there's no trick questions. Either you know it or you don't. And I think they maybe give you an hour, hour and a half. And I remember finishing that thing in, like, 10 minutes. And when I walk down, you know, you wait, the dial up goes, and they calculate your score and say, hey, you passed. And then you walk out and you're like a certified. And that was, like, the first time in my career that I felt like, oh, so if you put the effort in, you can gain the certificate. And when I got that certificate, I remember there was, like, a job fair where, hey, anyone that has A plus and Network plus certification, you can be part of the contractors that were replacing people's dial up with DSL at the time.
- [09:08]
Interviewer
Okay.
- [09:08]
Kelsey Hightower
And so that's how I, I guess, officially got into tech.
- [09:11]
Interviewer
Is it fair to say that you saw that this could be the most efficient way to get into tech at the time?
- [09:18]
Kelsey Hightower
I think I said I saw it as the only way.
- [09:20]
Interviewer
Why was college never, like, telling you, like, okay, that could be a way? Was it just. You didn't see examples?
- [09:26]
Kelsey Hightower
Yeah, I never saw the examples. I never saw the end game. A lot of the stuff that they were teaching, the curriculum, it didn't make sense that you would pay all that money. You know, look, maybe it wasn't a good school. Maybe it was the wrong class that I took. There's so many factors that could have went into this. But when I looked at it, none of the people that at the time that I was looking up to, this is not the path that they seem to be taking. And so I had enough of school. Right. If you're 18 at that time, you're like, look, that. That's enough of this. Because at the time, I kind of felt school was this. Because it was so easy for me, actually, you know, it's like it was easy to get straight A's. I didn't feel like there was a serious challenge. So it's like, hey, I want to go and do four more years of this. And I would later learn that, look, bachelor's is a lot of the similar that you go through K through 12. You kind of remember stuff. You listen to the lessons, but then masters, you challenge the material. And of course, if you make it to PhD, ideally, you're adding something new to the field. And I never saw anyone that has made it that far. So I never put that in part of my calculus. So. But just having that immediate feedback loop of, like, getting this A plus certification and feeling like, oh, I'M ready to participate in the actual economy, the ecosystem. So to me, I was like, this seems like a better path. And it felt like a path that I would control.
- [10:42]
Host
Yeah.
- [10:43]
Interviewer
And then what was your first job that you could get with, with the certification? This was the compta, right?
- [10:47]
Kelsey Hightower
Yeah. So at that time, Bell south was the biggest telco probably in America. They had been broken up by that time from the AT&T days. At that time, the people who did phone lines, right, so those are the official bellsouth technicians. They drive the fancy trucks. They have all the equipment. And when they made the shift to high speed Internet, that means you had to actually touch the computer. And I think as a union, they're like, look, we don't touch the computer. We don't even go into the house. We, we get to, you know, the D mark and we stop. And so they had contractors come in and the contractor's job were to come in, had to do a little bit of wiring. So if you had to run some cable, you did that. Create Cat 5 cables. You did that.
- [11:30]
Interviewer
Ooh, yeah. One of my first jobs was actually cabling, so I still, I forgot the exact color code.
- [11:35]
Kelsey Hightower
Orange, white, orange, green, white, green, blue, white, blue, something like that. It's burning your hand now and, you know, so you did whatever it took. And the other thing you had to do was you had to open the computer, you had to make a decision, right? If they had a new enough computer, they can use a USB modem. Those were terrible. They always broke and you would always come out for a repair. But for a new install, if you really wanted to do a good job, you install a NIC, right? Cat 5 port on the back of their computer. And at that time, like we're talking Windows 98. And so usually, I don't know, 20% of the time, as you're installing the drivers, the computer would crash. And now you have a whole nother situation. You have to now troubleshoot getting this thing back online or back operational. But if everything went smoothly, they now had a network card and then you had an external modem that then you connected to, you know, the phone line. And they had this high speed Internet connection. And then you connected the network cable. And I did that for about, let's say, a year. And then I started doing the businesses. So you go into people's homes like you're going door to door. And then when you go to a business, you would hook up one computer, but there's obviously eight computers there and only one of them has Internet access. And I remember at the time, you know, the business owner, it could be like a small insurance company, and they would say, like, hey, how do we get all the other ones online? And the first time someone asked me that, I'm like, I don't know, man. We put it on one computer.
- [12:53]
Interviewer
Like, pay grade, right?
- [12:54]
Kelsey Hightower
Yeah, we make sure it works. But then I decided, like, well, let me go learn. And that's when I remember, like, going to, like, Office Depot, where they sell computer equipment and things like this. And I went in the store and I asked him. It was like, oh, you can get one of these linksys routers, right? The infamous blue spaceship looking linksys router.
- [13:11]
Interviewer
Still remember them.
- [13:12]
Kelsey Hightower
And those things were probably like, 50 bucks. And I remember just buying one and figuring out how to get multiple computers to use one connection. And so eventually I was like, look, I can't do it as part of the job because we have to do this and we have to leave, but here's my card, and then they will call you. And I remember one of the first installations that I did on my own, they wrote me a check, and I was like, yep, you could just write it out to Kelsey Hightower. He was like, no, we don't write checks to people. We write checks to companies. And I remember right there on the spot, I'm like, oh, man, I need a company name. I just made one up. Digital Gateways. And they wrote that on the check. And I'm sitting there like, so how do you cash this? So I went to the bank, and they're like, sir, you have to get a business license. The business license. You could just do business ads. You have to do this. I'm figuring out now I'm 19 years old, like, okay, I got to go get a business license. Have to figure all this stuff out. So I do everything. I open a business account just so I can cash the check. But at that point, I'm like, oh, this pays more than this does. And so I got really good at doing those network installs. I really got good at troubleshooting because sometimes someone gave them a USB modem, lightning comes. The USB modem is fried. And then you would swap them out for a network card. And eventually I decided I can probably do my own business. And I decided to get some office space. So I opened a small computer store right outside of Atlanta. I would buy parts from the distributors. I was just like, 19, 20 years old, and I wasn't buying enough to really qualify for an account. But luckily, one of the smaller computer stores I used to buy parts from. Gave a recommendation to the distributor. It's like, hey, this is our guy. He's just getting started. And they gave me an account, and I was able to buy motherboards and GPUs. And people would come over and, like, they'd bring their kid and they would have a parts list. I want a computer with all these things. And we would assemble machines and sell them. But also it was the headquarters for all the other service calls. So I did that for, like, three, four years. That ended up evolving into. At the same time, now we're talking like 2000, 2001. A lot of the music studios were moving from analog gear and the large mixing boards to pro tools, right? The little rack mount unit. And they all needed max. They all need these conversions. So I added that to my abilities. And then I had a small setup in the store, and artists and musicians would come in and say, hey, we want exactly that in our studio. And I would get the order, and I added it to the list of things I could do.
- [15:42]
Interviewer
I mean, at this point, you now have your small business, sounds like it's going well, and suddenly you take a job at an employee job at Google. When I look back, how did that come up to now, it's almost like this is like, you know, the story oftentimes will continue. You become an entrepreneur, you grow your business, you just take it from there.
- [16:00]
Kelsey Hightower
Even during that time, as that store owner, I managed a comedian, and we went on the road. And so you were.
- [16:05]
Interviewer
You were helping out your manager?
- [16:07]
Kelsey Hightower
Yeah, I had a buddy from high school. He was a comedian. Turns out he was actually really good at it. We even. I went to go see him at a club. He's like, hey, I need a manager. I was like, are you even funny? Like, I know you from high school, but I don't remember you being, like, funny enough that I would pay to see you tell jokes. And he was like, I have a show tonight. And so I gave him a ride on the north side of town. And the interesting part, it was a predominantly black audience in Atlanta. Okay, makes sense. And he did these jokes, and they laughed. And it was one of these comedy clubs where if you're not funny, you have like, three seconds and they will just boo you off the stage, and that's the end of it. And he held it. I was like, wow, you survived that. That's incredible. And you were pretty funny. And then we drove about an hour north, and the audience is predominantly white. And so on the drive there, I'm like, there Is no way you can do those jokes in this room. I gotta see how this is gonna go. And at the time, they're only paying the comedians like $50. So you don't make a lot. In the early days, he totally pivoted the set and he held that audience too. And I was like, all right, I can be your manager. I know business, I know kind of logistics. I know how to, you know, make a plan together. And I did that for a number of years. And, you know, he won some televised competitions. We went on the road. Bands like Earth, Wind and Fire and some large comedians from Kings of comedy and Queens of Comedy. And I actually picked up some IT work with the company behind it called Latham Entertainment. They had been doing movies at this time. And so now I'm like, you know, doing it for them. And I got the small business, I got the comedian. And so look, I was able to save a lot of money, but man, Luckily I was 20 years old. Cause I had all this energy, but I was working quite a bit. Eventually you settle down, you get a family, and you do the math. And it turns out people kind of over glorify entrepreneurship. I think a lot of people believe there is tremendous upside, right? The type of entrepreneurship we talk about with software companies, the upside is crazy. But when you're doing like selling parts or service business, unless you plan to open lots of stores and, you know, grow a larger employee base, it's not the same growth trajectory as software companies. And so I kind of did the math. After four years of doing that, I said, look, I want to settle down. And if you've been an entrepreneur before, you know, employees get paid first, the owner gets paid last. And there are months where you get paid last or you don't get paid at all. And now you're kind of drawing from savings because it's not their problem. But I did well overall, like I did very successful. But I remember it's like, you know, I think I'm ready. And so I looked around and Google had data centers nearby. And I felt like I had a great combination of skills. I understood, you know, the racking stack part of the world. I understood the physical part of the networking stack. I understood everything from Linux to Windows. I had an entrepreneurial mindset. I didn't think there was nothing I could not do. And I remember going to that interview and they were hiring like data center technicians what it paid. And in my mind I'm like, that. You only have to work for eight hours. You don't have to issue Any invoices. And you get paid every two weeks, no matter what. No inventory. This is crazy. No way they're doing this. And so I go and I remember doing that interview, and I didn't know Linux that well. And luckily for me, I knew FreeBSD well. And as I'm answering the questions, I'm like, look, I am not an expert on Linux, not the way Google was asking these questions. And it's like three people on the other side of this table just rapper firing. And I remember I was like, I know FreeBSD. I swear I got lucky. This one of the interviewers, I think they had a FreeBSD tattoo on their leg, the little Beastie logo. And I saw this logo. I'm like, oh, I'm saved. And so we started going down the FreeBSD questions. And I pass, and I get this job in this data center. And it was good because it's like, hey, I'm working with my colleagues. But it felt a bit slow because you only get one job. You come in, you do this thing. And I got really good at it because to me, I kind of saw it as like a bit of a competition. Who is the best data center tech here? What are their metrics look like? How do I exceed their metrics? I want to learn how to do every particular thing in this data center. Because previously, as a business owner, the more skills you have, the more money you can make. And then I just started switching jobs every three to six months because I just wanted to explore everything, to just amass my abilities and doing the math. I think every jump was like a 25% pay raise. I mean, coming from a small base, it wasn't. It didn't feel big at the time. But after a few jumps, my salary doubled.
- [20:40]
Interviewer
Puppet was a bit of an inflection point in your career.
- [20:45]
Kelsey Hightower
You know what? And I would say the biggest inflection point in my career was there were two of them before I get into Puppet Labs. Okay, so you come from Google. You see this huge operation. There's hundreds of thousands of servers. The cables are perfect. They're immaculate.
- [20:59]
Interviewer
Oh, by the way, can you help us imagine, like what a data center back then look like at Google? And what did you do as part of the job?
- [21:07]
Kelsey Hightower
Yeah. So in 2000, maybe four, maybe 2005, that data center is like a warehouse. I mean, it's huge. So think about a place where. And I'm pretty sure they always exaggerated the numbers. So exaggerated number to think about is like, think about 200,000 servers in one place, everything is immaculate. So a lot of people have worked in data centers and it's a mess. Wires are all over the place. You know, you're ad hoc adding and removing servers. But Google was systematic. Those machines came off a truck, they were wrapped perfectly. When you wheeled them into their spot, you connected the network cables, they would pixie boot, they would burn in. So part of the job was you walked around with a crash cart. So depending on what your abilities were, some people had pretty straightforward jobs. You had a crash cart, had all your tools on it, and you would walk around and you would find machines that were needing of repair. So if you have 200,000 machines, it's okay if like 300 of them are broken. Right. The system can route around that and but if they were broken, they needed to be repaired. So you would go to rack a server 7, you will pull it out and you will look at it and say, oh yeah, the SATA card is on fire. Like it's literally burned. It's burning. It was burnt. And so he was like, I can diagnose this one with my eyes. It needs to replace the static card. But the thing is, you would go into the system and you would say, this SATA controller needs to be replaced. So you would be making your prediction and then you would replace it and you would go through its burning process, it would join back to the fleet. And then the way you were measured was how good were your predictions? Oh, right. So you said it's a SATA controller and to me, moving fast, you look as if I think that SATA controller. I'm not going to waste any more time on this one because I'm trying to get my numbers up. So I just swap this out of controller, you bring in all the cables, you slide it back, it goes through its process and you move on to the next machine before you get the feedback. Yeah. And fun fact, there's a guy named Tim Hawkins, very popular in the Kubernetes community. He's like the network lead. So back then, Tim Hawkins is working at the other side of Google, like the bigger side of Google. And one of his first projects was a little tool that you would put on the motherboard. It had about nine lights on it and the lights would flash back and forth and it will tell you what DIMM slots were potentially bad. The technicians that were fast, they didn't waste time like running a program to do a extensive test of the memory. You learn how to use this little device and you would walk up to the motherboard so the thing I would do is, like, before I do anything, put this on the motherboard, get the reading. And I learned to trust it over time. Then one dim, two are bad. And the goal would be, all right, I have all this memory on my cart. You swash those two DIMM slots, make sure that that's done first. Reboot the machine, and you might be like, I think that's the only thing wrong. And again, you were put in the system. I believe I only need to do DIMM 1 and 2. And then the way you were measured is how long before that machine gets kicked back to repair. So if it doesn't get kicked back within, I don't know, 30, 60 days, you did a good job. If you didn't, your scores would be low. So for the technicians that were just reckless, like, they wouldn't even try, and you're just swapping the wrong part. You're swapping the wrong hard drive. And so your stuff is always coming back for repair. You were not efficient. So I got to the point where I can maintain high 90s, but also repair, let's call it three times more machines than other people. So lots of machines, rate of return. And then I learned how to do the network switches. And there's power audits where you're lifting up tiles from the floor and you're making sure that everything looks good. You gotta be careful not to touch them because you could die. And so you learn everything about a data center, like the service loop. You're running all this Cat 5 cable, and it has to have a perfect service loop. Fiber runs on a different part of the rack. Right? So you don't ever mix these things. So as a person still, like, I'm in my early 20s, I'm thinking this is how all data centers look.
- [24:59]
Interviewer
It's not the case, but it's crazy because just as I think back, you know, I was a manager before as well, and of course, a software engineer for a long time. But the way your performance was continuously measured and fed back to you and you were evaluated based on it, it feels way more strict, should I say, than, you know, like, folks who work as software engineers, including at Google. I mean, the frequency, the expectations, the
- [25:22]
Kelsey Hightower
reason why I didn't feel as bad as some of the metrics people are using now is because I felt like I can control the outcome. It didn't feel like it was a thing. That was just a metric that didn't do anything. If I felt like my score was taking a hit and I was like, you Know what? I am being a bit sloppy and how I'm diagnosing these machines. And I remember one time where I almost had my score dip below 90. I started writing additional shell scripts to start combining different functions together. It's like, you know what, I can't be moving this fast. There's a way for me to diagnose multiple things in the machine at one time. And so I would diagnose the SATA array, all the hard drives while I'm doing the memory component. And then when I reboot it, I would just run the script one more time as I'm putting my cart back together to catch that one more thing. And once I started doing that, I can move as fast as I want it and the scores are right. So to me, when the scores actually match the things that you're doing, then it's a healthy feedback. And again, no one really talked about it unless you needed to talk about it. And so I kind of leveraged it for a personal thing. I pulled it up in the morning, I kind of looked at my performance metrics, and in many ways I calibrated my strategy based on this detailed feedback that I was getting. So I think I appreciated that level, that granularity back then, because it felt like it was something that was helpful for me, not just my manager.
- [26:43]
Interviewer
And you said you had two, two kind of big, big inflection points.
- [26:46]
Kelsey Hightower
One of them was Google was a big one. Like, that was definitely one, of course, my entrepreneurship. And when I got to web hosting, I went to a company called Pier 1, their spinoff of Rackspace, and they were all about fully automated self hosting. Right.
- [26:58]
Interviewer
So this was back in like, what, 2005, six, seven?
- [27:01]
Kelsey Hightower
Yeah, this is like, yeah, 2005, 2006.
- [27:04]
Interviewer
Oh, they were already fully full automated, full face.
- [27:06]
Kelsey Hightower
That was their thing. Their tagline was latency kills. And so back then you would go online, a lot of the customer base was like people hosting their own game servers, right? So if you wanted to play a game, one thing you could do back then is host a game server. But you needed a game server that multiple people could hit. And so you would go along to server Beach. So this is a spin out of Rackspace. So Rackspace is more like, you know, we'll buy a server and once you get it, and it's a lot of manual steps. Rackspace is more of a let's automated everything. So the machine would pixie, some PHP scripts would run if you ordered a RAID setup, then we would configure the RAID while The machine was NET booted. And then we would put you on the VLAN that you belonged on. We would install the right operating system based on what you've ordered. And we just took a form, we just went through all of these steps. And then when we're done, and it took maybe about an hour, when we were done, you had an IP address, login credentials, and if you wanted email and plus, you know, website management, all these things, you got it. And when you were done, you gave it back. And then we put it back into the pool ready for the next customer. And so when I saw how we were doing that, I was like, yo, this is okay. You can automate things end to end. And the other thing that was important here, we were doing things like updating the firmware for RAID controllers. Because once you pixie a server now you're in memory and you have access to all the hardware. You haven't committed to an operating system yet, but you have enough to do whatever you want. So if you want to configure the RAID controller. And back then there wasn't like clean APIs, we were literally running curl scripts and command line utilities, trying to get this machine into the right shape. And then when we were done, we would put it into the fleet. So at that early age, I'm like, oh, there is nothing you cannot do. When we had to automate the Windows 2000 servers back then, we would just build tools that would literally screen scrape login to active directory, and then we would screen scrape mouse movements. So that way we can patch software on those Windows servers. So the concept of like, I need a specific tool to do is like, no, Back then it was like, you do whatever's necessary. Because these people are only paying like $99 a month. They don't have time for you to spend a whole week. When they call random customer, you don't know their setup, you don't know their infrastructure. You have minutes to get them back online. So everybody learned to move quickly, no complaining. When you get that ticket, it's on you to figure it out. Maybe you link to your teammates for help. So I kind of learned how to move fast. But the inflection point came from, I started that job in tech support so people would call MySQL isn't working. DNS isn't working. They can't even describe what isn't working sometimes. And so I realized that we were all in the phone queue. When the phone would ring, it would just round robin between everyone. Then if you couldn't Solve it fast enough or you couldn't solve it at all. You created a ticket and the ticket sat there. And then ideally you get to it later, but the ticket queue would just get long. And when a shift change happened, we just had all these tickets piling up. And of course, customers are now mad. Three days, no response. So one day I said, look, I'm just not going to log in the queue, I'm just going to resolve the tickets and I'm back in that entrepreneurial mindset. I'm building little scripts to take a ticket. I see the issue. This is MySQL issue. We need to vacuum the database. This is easy. Run that one ticket. Close. Hey, sir, everything is good to go. Please try again. Close up. This is not even an issue. Close. Plus, oh, you can upgrade php. Close. And then the ticket queue is zero. It's just empty all day. And so someone's like, hey, Kelsey's not logged into the phone queue. He's not even on the phone. He's not taking calls. My manager pulls me in the office like, hey, Kelsey, why are you not on the phone queue? I said, because we don't all need to be on the phone queue. We can just have one person making the ticket queue stay at zero. And then I would tell my colleagues, hey, look, if you can't figure it out fast, just open a ticket super quickly. I will take care of it. So some people specialize in Windows. They got a Linux call, they didn't know what to do. I said, don't worry about it, just put it in the queue quickly, move on to the next call I got you. And so I became super efficient. So I explained this process to the manager and he thought about it. His name is Mike. And Mike was like, yeah, I like that. We're going to change it. We're going to have a couple people stay out of the queue. But the promise is that queue has to stay zero. And it's the first time I learned the difference between activities and impact. Activity is you being an all star, answering a bunch of calls. But the ticket queue for the team is still high. You making this jump and saying, look, maybe I stay out of the queue. And my promise to my colleagues is impact. The ticket queue is zero. So on Wednesday, when we do the team turnover, we're handing off an empty queue. Now, of course, I didn't teach them that because I didn't think about it. And when we would come in, the queue would be high again. It's like, yo, whoa, this is not you guys. Keep handing off a bunch of burden to our team, and of course we would clean it up. But then the management team is like, yo, everybody's going to do it. And then I was like, man, you can change the process. So that was like the huge inflection point. I think the next one was more about being a mature person. Stop job hopping so much. Because again, after maybe a few promotions and some impact, I'm now off to the next thing. And I didn't have to necessarily live a long time with some of those decisions or, you know, stay long enough to really impact more of the culture. So when I got to financial services, it was the first time I got restrictions put on me, right? Because in these companies, everything is about moving fast. Google, we got to move fast. We have to compete with the big guys. We're going to do with cheap hardware, smart people, any means necessary. Web hosting, we're not charging a lot. Margins are thin. You got to move quick. Financial services, like, no, we making money over here.
- [32:43]
Interviewer
You joined the financial services company?
- [32:45]
Kelsey Hightower
Yeah. So that was the first time my salary doubled. So I went on my lunch break for a job interview.
- [32:49]
Interviewer
That must have felt awesome.
- [32:51]
Kelsey Hightower
Oh, my God. I remember my manager was so upset I went on a lunch break.
- [32:55]
Interviewer
The previous manager?
- [32:56]
Kelsey Hightower
Yeah, I met Pier 1. He was a good friend of mine named Joe Rodriguez. He's the first person that made me a software developer. So I showed him some ideas I had on modernizing our virtualization, optimizing our PXE boot process. And he was like, man, you have a lot of good ideas. So in the interview, he's like, show me what you built. I'm so excited. I'm still that entrepreneur thinking. And I got the job. So now I'm a software engineer working on this automation stack. And I remember seeing jobs that I used to be afraid of five years ago. The same jobs that made me want to just go get an A certification and open a computer store instead of even trying to. And so I looked at those job descriptions again, and it was like, you need Lennox. I got that even at that job. I got Red Hat certified, right? So I was like, I got all the qualifications now. And I remember the job. I didn't know how much it paid before I went. But you had to wear a shirt and tie. It was the first time in my whole life I had to wear a shirt and tie.
- [33:52]
Interviewer
Oh, so you have to go and buy one?
- [33:53]
Kelsey Hightower
Yeah. And I remember my home EC teacher, she hemmed my pants for me, right? I was like, hey, I got A job interview. Does this look right? I'm driving up on my lunch break and I get there and you know, this is enterprise. I'm like, wow, this is, this is the big league. I didn't even think Google was big league. I thought financial services, big league. Obviously, if you have to wear a tie to do your job, you must be doing something so serious that you need a shirt and tie to do it. And so I get to the interview and I'm sweating like they're going to ask me stuff that I've probably never heard of or seen before. And they started asking Linux questions. I'm like, no, you got to use this flag. That's not the right flag. No, look it up. That is not the right flag. You can't do that with grep. No, you got to pipe it this way. The PS table doesn't show you that they're on that version of Linux, not with that kernel. And I'm like, it's too easy. And part of me is like, either I'm really good or something else. And so I'm thinking like, yo, that was. Maybe there's another round. And so I'm driving home and I'm loosening up the tie and I'm calling my wife like, hey, I think I did a really great job. Like, you know, and this has been a pattern. You know, I get good at something and I find a better job. I got good at something, I found a better job. But now it's like, this is like a career. And so I'm talking to my wife and I was like, hey, I gotta call you back. And it's the recruiter calling. Hey, you have a second? It's 100%. They want to make you an offer. I'm like, you know, make you an offer is different. Like, it's usually this is how much the job pays.
- [35:28]
Interviewer
That's how it was until then. How much were you making?
- [35:31]
Kelsey Hightower
45k.
- [35:32]
Interviewer
So you were making 45k at that point?
- [35:34]
Kelsey Hightower
$90,000. I almost drove off the road, like, double. This is insane. I'm thinking 90 and you're doing the, you know, retirement calculations. We can buy a house. You're thinking of all of these things that you could do. And so I'm on the way back to the job and I remembered and they were like, when can you start? It's like Thursday. I'm telling them we can start Monday, like two week notice. I'm like, I don't know, man. Like, it's double. I have to. So I remember I called Joe. And Joe's based in San Antonio, where the headquarters of RecSpace was. I was like, hey, Joe, man. Hey, man, I gotta quit. He was like, what, man? You just. You know what I'm saying? You're doing so well on the team. How you gonna quit on me? I made a bet on you, you know, I said, hey, Joe, calm down, bro. They pay 90k. He's like, what are they hiring? And so I got to. I got to that company, Total Systems Thesis, and it did feel a bit slow. They had run books, change windows. Everything was regulated. This is a financial institution. We're processing credit card payments. We're doing work for the government. We're doing all of these things. And I remember the team was just. Everybody just moving at this pace.
- [36:46]
Interviewer
But isn't it crazy when you get in there? I had something similar when I moved into London, and my salary more than doubled. Going for Edinburgh, where you have all these expectations because you're being paid so much more to tie. Same thing for me, for the tie.
- [36:58]
Host
And then it's a bit disappointing.
- [36:59]
Interviewer
And I think, did you not feel
- [37:01]
Host
a little bit like, am I missing something?
- [37:02]
Interviewer
Like, this should be higher. This should be. The interview was okay. Maybe not as hard, but it was something.
- [37:08]
Kelsey Hightower
I mean, I did because I was naive, because I didn't understand the consequences of a mistake. And so, yeah, I was like, oh, these people are just moving slow. And I looked at what they were doing, and everything was like a risk. If you made a mistake, you remember you only get 7 milliseconds, 7 seconds to give Visa response. If you don't, then it's a default decline. Now everybody's losing money. And so the cost of getting a good change, it was just worth waiting. So I didn't know that in the beginning. So I'm just like, oh, everybody's moving a little too slow. I was seeing how they were doing deployments. I'm watching how they're provisioning servers. I'm like, you know, you can automate this whole thing because I've done it multiple times. And then I learned to be a little bit patient. So. All right, hold on. I learned how to deal with the no's. I learned how to deal with the executives. I learned how to talk not just to engineers, but the senior leaders and get their trust. And I won't talk about everything that I did there, but I was there for about three years, so the job hobby stops. And I remember there was a task where we were using Apache and some Java plugin to talk to our JBoss instances. And so they were using a lot of memory per connection on a load balancer. And so we got a new. We were moving off the mainframe, we were moving into this new Java world and the servers just kept falling over. We couldn't handle all the transactions. And I'm just watching the team go into these change windows and then we fail. We would fail, we would fail. And the CTO at the time was like, you know, this is costing us money. We're getting chargebacks, we're having to pay out penalties. And I was like, I had a dev environment where I was using nginx. I got rid of all this Java stuff. Like it's just HTTP, you don't need a Java connector thing. I'm reading the spec, you don't need it. And my memory usage is a fraction
- [38:54]
Interviewer
by getting rid of it.
- [38:56]
Kelsey Hightower
Yeah, you don't need it. And nginx had a better threading model, all these things. And I was like, I got a perfect config. I ported the Apache config to this one. I even handle all of our redirects, all of our routes, all the legacy cruft. This thing will work. It's like, I don't know, Kels, this is not certified stuff. It's like, no, no, no. I got it from Red Hat. RPM install nginx. It's in red hat. And they're like, okay, it's in red hat. Okay, sure. It's giving me one chance. Yeah. And about after a week, they gave me a shot and they let me be in the change window.
- [39:26]
Host
And what is the change window?
- [39:28]
Kelsey Hightower
So change window typically in the financial institution is like, we can start changes at midnight and we have to be finished by 6am and you have to notify every customer we're going to change something in the environment. Things may get a little weird. We would like your permission. And if it's something that can be detrimental to your business, now's your time to speak up. So once we got permission, we had a window. That's the only window you got. And if you can't finish it on time, you got to shut it down and get us back to where we were. Yeah, rollback, roll back, roll forward, or if you could, roll back. And I remember I put NGINX in place. So at this point it's all you. No one at this company has nginx experience. Also, most people don't want to do this. They're like, hey, we think this is a bad idea. And so now it's your Reputation on the line. There is no hiding behind the team. It's almost like everyone's sitting back, like, you got it now. Luckily, the leadership was like, we are supporting you and we're putting our careers on the line as well. I mean, they probably would have been safe, but after about two hours, I got everything working and we just watched your memory pressure just drop. So if we were at 90% on utilizing, let's call it 32 gigs, and we're just blowing the stack every time, everything's crashing. This thing is hovering at like two gigs of ram and everyone's like, yo, like, are we getting peak load? Is this legit? And we're all just sitting there, like, not sure. And the way we used to test the platform is we would. Someone would drive to the gas station, get a credit card, buy some gas. We wanted to see the transaction land in the Oracle database.
- [41:01]
Host
Yeah, you could track it so we
- [41:02]
Kelsey Hightower
can track the whole thing. We're like, yo, it works. And so the thing is, you can't really be comfortable until about 10 o' clock the next day. So now it's 3am, we all go home. I don't go to sleep. I'm like, yo, we've changed a big part of the infrastructure in production. Let's see. So 10 o' clock goes by and we look around and there's no nagios alerts going off. I'm like, man, we might be good. You know me, I got top going. I'm looking like memory usage is holding steady.
- [41:33]
Interviewer
Yeah, top is for those who don't know that. That's when you showed the CPU, you
- [41:39]
Kelsey Hightower
PS3 in a loop. Just like if you ran the PSAUX command, you see all the time. Oh, yeah. You see the processes. Yeah. And so you just run top and then it's basically that every second being refreshed or whatever timer you give it. But one of the bets I made was if I get this to work, you have to buy pizza for the whole floor for a week. Every day we get a new order and. Cause he promised, like, kelsey, if you get this to work, I'll buy you a steak dinner. And at the time, I still am. I was vegetarian, so I don't eat steak. But here's what I would like instead. You gotta buy pizza for the whole floor. Now, I don't know how strategic I was thinking, but I figured that I could score some points with the whole team from turning it into a I succeeded to a we succeeded. And then people eating together was one good way of doing it. And I remember after it worked, he was like, what's the order? And I just ordered, you know, pizza from one place and we just put it in the middle. Everyone was like, what's the pizza for? Right? Cause usually you only get this for special events, like for the thing we did. And the next day some more pizza, and some more pizza. And so when I got that accomplishment, I was like, okay, this is what maturity looks like. It's not about just having the right solution. You literally have to get consensus, buy in, make sure, and then your reputation's on the line. And I got that change in and it just kind of changed the way I thought about what success looked like. And so that was the inflection point. And then of course, I brought Puppet into that organization, repeated a similar process of automating everything with this tool.
- [43:06]
Host
Kelsey just talked about what CICD looked like in the 2010s and how difficult deploying to prod was, which leads us nicely to our episode sponsor, buildkite. This type of manual deploy error was the time of buildkite was created. Engineering orgs were trying to figure out how to ship more frequently more than once a week. And those running into Wal first turned to Buildkite. Shopify started running on Buildkite in 2015. Airbnb, Canva, Uber and OpenAI have all been using Buildkite for seven to 12 years. Throughout the 2000 and tens, the hardest CI problems in the industry for global software leaders were getting solved on buildkite pipelines. That's exactly what makes buildkite the CI platform most in tune with what's happening right now. Genai Throughput is pushing everyone's BuildQ to its limits. Coding agents are pushing commits 5, 10, 50 times the volume that your defaults were built for. The same architecture that absorbed Shopify's traffic at scale five to ten years ago now observes the share push from Anthropic Cursor, Meta Mistral, Cohere, vlm, XAI Lambda and more, who are all Built Kite customers. Where other CIs are cracking under the weight, Buildkite is running about 1.4 billion job minutes a week. Agents run on your infrastructure or on buildkites. Every artifact and log is captured and when something fails, you see the why immediately. If you're hitting ceilings on your existing CI or downtime is becoming more problematic for your Delivery. Head to buildkite.compragmatic 30 day All Access trial, no credit card and an actual human NGR on the standby. His name is Ola and He is very helpful. I'd also like to mention our seasoned sponsor, Sentry. AI agents don't have a feedback loop from production observability logs. But should they have it? I mean, we probably don't want to build a system where after each production error, AI agents automatically push out a fix into production without supervision. That would be a recipe for disaster. But what if we did something more modest? When a production error fires, an agent investigates this with context from Sentry. Sentry already has all the context in the error. After all, Sentry MCP is one way to plug in Sentry to agents that
- [45:04]
Interviewer
support the model context protocol.
- [45:06]
Host
Claude code, cursor codecs VS coded copilot. They all do. After you hook up the MCP server, you can do some very useful things. For example, you could do this. When an already resolved Sentry issue resurfaces. You can kick off the cursor agent to investigate the regression, read the relevant code and open a PR with a suggested fix. There's a little work involved to get all of this going. You need to connect Sentry to your code repository, add Sentry MCP to Cursor, define the instructions for Cursor's agent to investigate, configure the trigger that launches the automation and test that it all works. But once you have it up and running, you can get regressions fixed faster. Whilst they're reviewing every and all fixes. This feels like a sensible and helpful use of AI, NCP and Sentry to me. Check out sentry@sentryiopragmatic and start monitoring and fixing regressions today. And with this, let's get back to Kelsey and when he introduced Puppet into his organization and automated DevOps processes.
- [45:58]
Kelsey Hightower
And I remember we had someone from
- [46:00]
Interviewer
Puppet come by and just for again, those who don't know Puppet, Puppet is
- [46:03]
Kelsey Hightower
this integration management tool. So Puppet is. We went from, you know, lots of shell scripts, you know, doing random things and pseudo automation, but most of the time you just did a manual thing. You had a ticket come in, someone wanted a new SSH key on a server, someone wants something installed, you did it manually copied and paste the output in the ticket and you closed it. And you did that every day over and over again. And that's where the saying that I say sometimes some people have 20 years of one year experience, some people in it have been doing that for 20 years on the row. They never made the leap to automation or learning new tools or skills. And so I saw like man, maybe I can bring in a tool. And I brought this Tool called Puppet at the time 0248. And this is right before DevOps becomes a word. But it's like, all right, I'm starting to learn how to write code like the. For the Java production stuff, I'm writing Puppet, you know, Manifest using the Puppet dsl. Sometimes you have to write a little bit of Ruby to build a new resource type. And I'm contributing and also I get introduced to open source, like, hey, there's something there that doesn't work. I'm going to contribute it upstream. So I'm doing all this behind the scenes, getting ready for Act 2. And I remember my manager went to a conference and he came back and was like, hey, Kelsey, I finally know what you're doing. I was like, what's that? He was like, you're doing DevOps. And part of me was like, upset. How do you get to name what I'm doing? And so, you know, everyone's talking about DevOps and he's like, I met a guy named James Turnbull and James is an Australian dude, he worked at Puppet Labs and he wrote the first Puppet book that I bought and read. And so I'm like, James Turnbaugh. That name sounds familiar. Oh, I'm looking at my desk. James Turnbaugh and Jeff McCune. So he's one of the co authors of this book. And it turns out James wrote a lot of tech books back then. And he's coming to the office and my manager wanted him to check out how we were using Puppet. I was like, okay, so you know, this is world class expert. And by that time I had Puppet hidden behind JIRA tickets. And so you could just open a Jira ticket. I had RPMs for everything. You picked the RPM you wanted from a dropdown, you picked your environment from a dropdown and magic happened and you got results. Most people didn't even know Puppet existed. Most people never touched it. They just knew they can get anything on demand.
- [48:19]
Interviewer
You could go to Jira and it does it. So for example, you could like provision a new machine or call environment.
- [48:25]
Kelsey Hightower
I want database, I want IBM MQ series message server. I want these three apps and I need this firewall setup. No problem. Open the right Jira tickets, get them approved. When they're approved, we had this little. I called it Mr. Resetti. One of my buddies gave me a name, is from some video game. Once it got approved, Mr. Versetti would own the ticket and we would just extract the custom fields and we would just call the right Puppet manifest and we would take the output and update the ticket. And so PMs developers, they just got what they want damn near instantaneously. So we built all the systems. They didn't know. Nights and weekends, I'm contributing to Puppet because, you know, wasn't allowed at work at the time. This is 2007, 2008.
- [49:04]
Interviewer
They probably didn't even really understand what open source was anyway, so they was
- [49:08]
Kelsey Hightower
like, hey, you gotta do that on your own time. So I would just do it nights and weekends. So James shows up, and I remember we all get in the elevator. We're going to the third floor. My manager's introducing James to me, to our team. Like, hey, this is this person. This is this person. This is Kelsey. And James turns to me. He's like, kelsey Hightower, oh, we know you. We love your contributions. And my manager's just like, what contributions? We're not doing contributions. And James is, like, shaking my hand, and he's just talking to me like, we're friends. Because the thing is, I have been working with the Puppet Labs team through, like, core open source, Open source, contributing to Puppet. And so we get up there and, you know, my manager's like, hey, let's show him our setup. I was like, all right, James, we have a lot of Puppet manifests over here. I'm using external node classifiers. I'm setting a Brighton config from Data, and I read Mark Burris Promised theory and got it all dialed in. But from an interface perspective, I just didn't want everyone to have to learn Puppet. So we hooked it up to Jira. I'm showing him this entire workflow, and he's just sitting there like, yeah, I have no recommendations at all. Like, this is. This is amazing. And here's the thing. That kind of the game changer. So he came all the way to Atlanta. We're 45 minutes from the airport. He spent time with our team, and it's the end of the day. And then he's on the way out, and he's about to call a cab to go to the airport. I'm like, no, you're in my hometown. I cannot let you go to the airport. I will give you a ride to the airport. So I get in my car, and he's in the front seat. My colleague's in the backseat. He takes off his tie and rolls up his sleeves. I can see all the tattoos. I'm like, oh, so you're like a legit. He's like, yeah, I don't do all of this. I just did this for you. All this is your dress code. I was like, oh, James is cool. He was like, man, yeah, you guys are doing great work. And now I'm talking to the real James Turnbull. And he invited me to give a talk at Puppet Cough. And I went to go give that talk. It was my first, like, real conference talk. And I talked about some low level kind of puppet integration stuff. And I had a little comedy in there. It was like the first time I felt like I could just be myself on stage. Before I came home, Luke Kniece, the founder of Puppet, we sat down at a little coffee shop in Portland, Oregon, and he was like, you know, would you like to work here? I was like, yes, like, of course, 100%. And of course, there was a nice salary increase and I could work remote. I didn't have to leave Atlanta immediately. And I remember coming back and my manager really appreciated all the progress I was making over the years. I kind of made a name for myself. And I remember he gave me a nice little raise just out of the blue. And he slid this envelope to me and I opened it and it was a nice number in there. I was like, oh, that's great. I was like, but here's the thing. I'm resigning. I'm going to go work for Puffett. And then he said something that was dope because we had a kind of, you know, difficult relationship, depending on how it went. But overall, he helped me mature. That's one thing I will always say. He helped me really be mature. And he was like, I'm surprised you stayed here this long, right? As, like, this ultimate compliment that you stayed here beyond the just getting better or making an impact. You literally changed this culture. And the full circle moment happened like seven years later when Kubernetes came out. I remember coming back to that office and they had already been running ETCD in production, Kubernetes in production. And even some of the old tooling I had was still running. And to me, that was like a really important feedback that sometimes if you make an impact, lasting impact is way different. That even when I wasn't there, they still had the culture to say, there's new technology and we know how to bring it into the stack.
- [52:45]
Interviewer
So at this point, containers were starting to start. Right. How did you first come across containers? And very early on, you realized, this is going to be big, big and important.
- [52:58]
Kelsey Hightower
I didn't.
- [52:59]
Interviewer
Oh, you didn't?
- [52:59]
Kelsey Hightower
No, because I'm at Puppet Labs. You know, before that, I was contributing to core Python stuff like virtual Int Pypy on the team that was trying to integrate some of the Python and Python had a package manager problem even back then. And so I'm all in on Python. And then when I'm learning Puppet, I'm all in on Ruby. At that time, I'm thinking DevOps plus configuration management is the end all, be all for everything. And really, at that time, we're talking 2011, 2012. In my mind, the competition was only between Puppet Chef and Ansel Ball. That was it. And so I'm all in and I'm working at Puppet Labs. And so we're kind of, in some ways, I felt like we were dictating the future. We were going to companies and getting them to understand configuration management. We were talking about the benefits of like speeding everything up and, you know, doing all these compliance. We were Moving people from SharePoint to executable code to implement these things. So we thought we were just getting started. There was no world where we didn't think that we needed 10 more years of that, right? We even got money from VMware to integrate Puppet into that Shift was being integrated into AWS at that time. So this was like the thing that finally, we're finally getting there. We just got to teach system administrators to become engineers via DevOps and start embracing these tools. Then the first thing that I saw come out was Golang. And I remember sitting at my desk and I was like, you know, at that time we were hitting performance issues with Ruby, right? The Global Lock interpreter. You can't do more than one thing at a time, not easily. And so we moved to things like JRuby. We even did closure for some part of the Puppet stack. And I remember we were thinking about, should we start doing stuff in C? Should we start doing stuff in C? But the problem is contributors. What would we do with all the Ruby contributors? Right? We relied on things like rubygems and it's going to be. It was going to be a hard sell. And I remember downloading Go for the first time and the thing that sold me on it, I remember one of the prototypes that I built was Factor. So Factor is one of the agents in Puppet that gives you facts about a machine. So this is Red Hat, this kernel version, this is what the users look like. And then we'll give that to Puppet so that your configuration code had context on what to do. And that was written in Ruby. And sometimes it will run slow because you're reading all these files serially. And so I remember I was like, all right, let's try this Go. Laying out. And I remember I wrote the code on my Mac, compiled it, SCP'd it to a Linux box. I was like, oh, this is crazy. And I remember running all the facts in parallel and it came back in a fraction of the time. I was like, yo, we gotta use Golang for stuff. I remember the team was like, nah, it doesn't work for Solaris or aix. Because Puppet was now moving into the enterprise. I was like, that's the criteria. No way. But I got it right. I've gone through this maturity thing before. And then Terraform comes out. But before Docker, Terraform is starting to challenge things a little bit, right? And it's starting to use things like Go. At least that was on my radar. So Terraform is like, who cares about the node? It's all about the APIs, and all of us come from the server world. And everything's about an agent on a node versus the cloud. Starting to take over at this point, right?
- [56:13]
Interviewer
Yeah. So Terraform was built with the cloud in mind, Right? You could configure your cloud environment infrastructure as a code.
- [56:20]
Kelsey Hightower
Yeah. And we were trying to do this from a node. We were trying to teach Puppet how to talk through by having this indirection where you have to go through a node to configure a server in the cloud. It's so weird. But then Terraform, Mitchell, Hashimoto, right, he was a big part of the DevOps movement early too. Then he was like, Vagrant was written in Ruby, so he was part of that same ecosystem. And then he splits off and there's this golang. I'm like, look at that. They're using Go for this. And I remember when Docker came out, Puppet at that time was the one pushing innovation. We're asking people to think a little different, get out of their comfort zone. And then Docker comes out and most of the office was dismissing this as like, nope, this is a fad. They don't even have config management in this thing. They don't really understand Enterprise. This is just kind of not like a toy. They were not being disrespectful. But no one saw this as a challenger, what we were doing. And I looked at it and I didn't get it immediately until I left. When I left Puppet, I built this tool called confd. I went to go be a VP of Engineering and I got to write Go code. So we started rightfully so. We lifted all the Java heavy usage and I earned the trust of the team. We started rewriting Some of the microservices in Go, shrinking our cloud footprint. And we sunsetted it and got it all into production, the GO code. And I was like, we don't really need Puppet anymore. And I open sourced this project called confd and it will pull variables from ETCD and generate just enough config. Just the parts of Puppet that I thought made sense. And then Docker was out. And then I was like, wow, we can probably stop moving Python files around. We can probably package them up, not in RPMs.
- [58:00]
Interviewer
The idea with Docker was right, that it would define a virtual machine or a container. Right.
- [58:05]
Kelsey Hightower
So to me, the big value of Docker at the time was previously I had definitely did the work to make RPMs for every app, even the custom apps we were using.
- [58:12]
Interviewer
Rpms.
- [58:12]
Kelsey Hightower
Rpms are the red hat package management.
- [58:14]
Interviewer
Yep.
- [58:14]
Kelsey Hightower
So if you're in a Red Hat system, you can do yum install nginx, yum install postgres. But most people, even today, don't package their third party apps like the apps that a development team would write. You're like, no, we just put cicd, we'll copy them over there, maybe put them in a tarball. But we usually never went to making official Debian packages or RBM packages that Puppet meant you didn't have to go through all of that work and you can still end up with a package, something that was repeatable. So all the stuff we used to do with Python and Virtual, all the things we used to do with Ruby Gems and the virtual environments we have for that, we got rid of it. Just squish it all into a Docker container and you got rid of a lot of dev tooling. So this is why I think it resonated so much with developers, because we cleaned up the mess of working on multiple projects. And so I was attracted to that. COMPD was compatible with that. And then coreos was the thing that I was like, you know what? I think I know what I want to do. But I didn't understand distributed systems, not to the degree what coreos was doing.
- [59:17]
Interviewer
So when I moved to coreos, what was the idea behind coreos?
- [59:20]
Kelsey Hightower
So coreos was like Google's infrastructure for everyone else. And so at that time we did. There was a tool called Mesos. The Borg paper had already been published. I tried to read it when I was at Puppet Labs. I don't understand any of this stuff. Mesos is hard to install. I couldn't justify it. But we would see kind of the rumblings of the Twitter folks talking about the distributed system and all this maybe big data stuff is going around. But I was like, I just don't understand it. It seems incompatible with the stuff that I think about. Even as someone who's worked at Google that had a system like this, right? They had Google cluster file system at this time, it didn't seem as complicated as this Mesos thing. So I was like. I dismissed it. But what CoreOS did was build on top of Docker. CoreOS is like, what if we had an operating system that only had Docker on it? Everything is written in Go. We can have a little key value store where you can put your config and we would just synchronize it to all the machines. And that was so compatible with the Comp D way. And so I'm looking at this Coreos thing, I'm like, yo, this looks a little bit more like the future, because that's what we used to do at this time. I was a software developer at Puppet. I was a software developer, VP of engineering at the other company. And I was like, you know what? Ops can learn a lot from the darker way of thinking about the world, because as a system administrator, we always try to make the OS small, remove things you don't need so it can be secure and repeatable. Docker was like, for the things that need to change, just put it here and isolate it. And coreos is like, what if you had an operating system designed only to do that? And I met the CoreOS team at GopherCon, a Go language conference, and they saw me present and what I learned from the Puppet days. Every presentation's an interview. I don't think a lot of people think about it that way because people are looking at you on stage and they get to see what you're about. And so at GopherCon, I remember also Rob pike and the original creators of Golden Garden. This is like the first Gopher Con, right? And so the original creators, Russ Cox, all of these people, I look up to, Brad Fitzpatrick, and they're all in the audience. And I remember I had a talk. Two things. I'm sitting in the audience, waiting for my turn. I'm number four on the list or something like this. And the two people who started Gopher Con, they're just from the community, Brian Kedleston and Eric St. Martin. And they're new to this conference scene. Like, you could tell, right? You know, they weren't the best MCs in the world, but they were like, hey, welcome to Gopher Con. And at the point in time, Ruby is still dominating, and down the street, there was a Ruby conference going on. And we were just in this other building. And I remember the first talk they introduced Rob pike, and he gave this amazing keynote around hello world. He went up all the way down to the compiler, he went all the way up, and he described how the language took shape. It felt simple on the surface, but it went super deep. And it's like Rob pike had also come full circle from his AT and T days, through Unicode, through all the stuff that he's ever done. And now we get to see one of his best works, Golang. And my talk was titled Go Lang for System Administrators. I wanted them to see that there's a better way. The way they introduced Rob pike, though, I was like, oh, man, this is Rob Pike. You gotta. You gotta have more energy than that for Rob Pike. So I'm sitting in my chair, and I don't know them, but I'm sitting with a buddy of mine from that company where we rewrote everything in Golang, Billy Kleeg. We're sitting next to him. I said, hey, Bill, I'm gonna go ask them, can I be the emcee? He's like, what? Okay, so I'm whispering and I walk to the backstage. It's like, hey, Eric, Brian, can I try my hand at being the emcee? And they're like, sure, right? And they knew who at least who I was, because they accepted the talk. And so I don't know if it was after Rob, but I came out next. So, like, hey, I'm Kelsey Hightower. You may not know me, but I'm going to be your emcee for the rest of the day. And I don't know what happened because it was my first time doing that. I'm cracking jokes, I'm having fun. And I said, hey, from here on out, when anyone comes to the stage, it needs to be loud in here to the point where we can get kicked out. And so I was, all right, so our next speaker, and I think it might have been Derek Colston, you know, he's like a googler. He was talking about some ghost stuff. And I remember it got really loud. And then they come on stage. If you've ever speaker ever before, and it's like you walk up to basically a standing ovation, you're energized, everybody is excited. I don't care what you do, the energy level is high. And so he comes out, he's having a good time. The audience is having a good Time. And then eventually it was my turn and I realized, like, who introduces me? So I go onto stage and I was like, I'm next. So we're going to try it like this. I'm going to go back and I'm going to come out, and you guys are going to make a lot of noise for my talk. And then I remember just like sliding slowly behind the curtain, so everyone's now laughing. And then the music comes on and I walk out surprised, like, hey. And everyone's clapping, and I do this talk. And at that time, it's live demo or nothing. And at the time, they had this thing called Go present. So Rob pike team, they wanted to make sure they could use Go for everything, even generating a slide deck presentation.
- [64:26]
Interviewer
It was all in Go.
- [64:27]
Kelsey Hightower
It was all in Go. So you had this nice HTML representation of your slides. You just open up your browser.
- [64:32]
Interviewer
Hopefully it won't crash.
- [64:33]
Kelsey Hightower
Hopefully it doesn't crash. And you can run executable code in your examples. So any code snippet you put there, it was formatted nicely. And you had the little run button so people can see the output of the Go code. And so I was like, hmm, what would a system administrator value you can get from go? How can I prove it to them? And so I wrote an ipixi server in Go. And I remember in VMware Fusion that runs on your laptop. So if you want virtual machines on your laptop, you can use desktop parallels or VMware Fusion. The thing about VMware Fusion on your Mac, you can create virtual machines, but you can also swap out the network card. And I did one where you can have a network card that talks to ipixie. So I wanted to show them how it boot up multiple machines from a Go Pixie server running on my laptop. So part of that demo, I was start the. I had this kind of snippet of my Pixie server and I hit the Run button. So now my Pixie server is running in the background. It's like, so, what can you do with it? And I remember looking in the front row and Rob pike and team were just intimately looking at this thing. Like, wow, this. He just started a Pixie server from his laptop from go. Present the slide deck tool. And so this thing is running, and I'm like, all right, let's bring up VMware. I got to make sure I switch the adapter to the one that supports iPixi. You got to do some firmware thing. And then I remember I booted the vm. You can see the logs in the go present of like HTTP handing off the image, giving an IP and the virtual machine is booting up. And you can just see the amazement on the audience face, like, did you just do that? And then I booted another thing in the thing and I'm going through why. I think that this is a game changer for the craft that we have. We finally have a tool where we can write high performing things with the simplicity of imperative programming and we can go beyond just scripts. We can actually build systems. The audience was dialed in and it got loud like after, you know, things were working, people were clapping. I look in the back and there's a whole bunch of people now standing and I recognize some of the names because they're from the Ruby community. These are people that are writing the Ruby books. And then, you know, after I'm done, there's a break and I walk back there, they're like, hey, we saw on Twitter that you guys are just like going crazy over here. It's out of controls, electrifying. We left the Ruby event to come here and so I felt like, man, it had arrived. But guess who else was there? The Coreos team. The Coreos team was watching me because I was pixie booting Coreos. This is what I mean is every job's an interview. The team was like, all right, we can see that. Yeah, you should join the Coreos team. So that's how I ended up at Coreos. This is how I really felt. Like the container movement had legs. And this is what, maybe a year or two before Kubernetes comes out?
- [67:14]
Interviewer
And then Kubernetes came out and you were starting to contribute to that one as well.
- [67:20]
Kelsey Hightower
Yeah, so I'm at Coreos and I think they think that's very important. A lot of software engineers, sometimes they look at the people on stage and they ask questions like, does that person know what they're doing? Is this person just like an evangelist? Did someone write this demo for them? Did they give it to them and they just run the code?
- [67:38]
Interviewer
Yeah, we think that. Don't worry.
- [67:40]
Kelsey Hightower
Well, I mean, I can see why, because sometimes it's hard to value skills you don't have. So a lot of software engineers are terrible. If you put some. They can't talk, they can't simplify concepts, maybe they can write code really well. But this is a set of skills that they may or may not have. And so when you see someone like that, you're questioning them. And I remember giving a talk one time at strangeloop about ETCD and CoreOS. And someone was like, do you understand? Is ETCD like a CA system or AP system like the CAP theorem? And the question was kind of loaded, like, we don't think you even know what that means. I was like, ETCD is going to always favor consistency. He was like, that's not correct. The Raft paper, blah, blah, blah. I said, listen, you said etcd. Let me show you. This is the RAFT log. This is my three nodes. I'm going to turn off two. And you notice I can't write any keys. So availability has been sacrificed. Consistency is being preserved. I'm going to start another node. There's going to be a handshake, there's going to be quorum. I will be able to write keys. Now it works. That's the CAP theorem in reality. So it doesn't matter what the RAP paper says. Are you talking about a RAFT log in a single implementation? Raft doesn't talk about cluster membership leader election, how it's implemented and what you should do in the different modes. That choice is yours. And this is how ETCD is implemented. And I remember he was like, oh, shit, this guy actually knows what he's talking about. Being at coreos, we were working on our own Fleet management system called Fleet. We were using systemd, and we were trying to synchronize configs through etcd. And remember, in a Coreos cluster, all the nodes communicate via etcd. So imagine using systemd. For those that have never used systemd, you put a unit file, like, I want this process to start with these flags, you know, bind to this port and you put it in a directory and then systemd will start it. And if you had a thousand nodes, of course you could SCP that file to all 1000 nodes and then there you have it. So we decided instead of you copying all the files to all the nodes, just put the unit file in ETCD. And the node that should run those things would then pull from ETCD and just run those unit files. And we called it Fleet. So we had our own vision of giving people a distributed spouse or distributed system. About a year goes by, kubernetes comes out, and everyone's like, what's that? And we were all, we got like a day notice. So the Google team reached out like, hey, tomorrow we're announcing this thing. Here's the GitHub repository. You guys are under embargo. Don't talk about it. And so I'm thinking, we're not part of this story. Like, our names are not in Here, it's just this Google and Red Hat, and they're going to announce it at dockercon. There's no core OS in here, so what can we do? And I remember I stayed up all night. I got access to the GitHub repository, reading all the go code, trying to figure out what all these binaries mean because there's no docs.
- [70:38]
Interviewer
Yeah.
- [70:38]
Kelsey Hightower
And so I got everything working on Coreos. So when they did the official Google announcement, of course there's a famous Docker keynote where they unveiled it to the surprise of even Docker to some degree. And I remember they post their number one on Hacker News, and then we post, hey, what they just said, but here's how you run it on CoreOS and some examples and commands. And I had to do a few patches to get it to actually work, and I had to build some binaries to get things glued.
- [71:08]
Interviewer
You did that overnight?
- [71:09]
Kelsey Hightower
Overnight. I didn't go to sleep. I'm just like, hey, guys, I finally figured out how to compile everything. I think the Kubelet does this. You have to put this there. They had like a Kube up SSH script, but I had to reverse it because I'm not using Google Cloud. We got coreos, we're on bare metal, so I have to pixie boot some things. But I think if you put all these things in the right place, there's this ETCD thing that's ours. We know that, so we know how to use that. But I think the API server connects to that. There's no volumes, there's no config maps. So all you can do is get this thing stood up, and then you can submit a config, and then it will just basically use Docker in the background. Okay, I can document that. So we get everything to work and I write this nice little guide. Someone on our team, they publish it to the official Coreos website. So then we launched that on Hacker News, and then we go to number one, and everyone is like, google just announced this thing. We don't know what it is. And then Kelsey launches this thing. We now know what it is, we know how to install it, we know how to run it. So now people are downloading Coreos just so they can play with Kubernetes. And I had a keynote probably in a week. I don't know what I submitted to the conference because usually you submit like months ahead of time. I was like, hey, I know this talk was supposed to be about this, but it's not today. It's going to be about Kubernetes and people like, what's that? It's like, yeah, I just got announced, I want to show you. And I started giving people live demos of Kubernetes and how to make it work, you know, using core OS and all of these things. But the team still wasn't sold because we had our own roadmap. Yeah. And also at the time.
- [72:34]
Interviewer
And you had your own Fleet management software.
- [72:36]
Kelsey Hightower
Yeah, we had our own Fleet management.
- [72:37]
Interviewer
And Kubernetes was now competing with Fleet specifically.
- [72:40]
Kelsey Hightower
Yeah. And also we didn't know we can trust it. Right. Remember, Docker is the king. Docker's number one. There's Docker Swarm. Right. And we're with Fleet. And right now we're like, Google also launched years prior, a thing called. Let me contain that for you. It was a container runtime written in C to compete with Docker. And no one cared.
- [73:00]
Interviewer
Yeah.
- [73:00]
Kelsey Hightower
And so we didn't know, like, maybe no one's going to care about this either. It's only when I started going home and starting doing small contributions and starting to read everything and getting the feel for it, I was like, no, there's something here. And so maybe two or three months. And luckily, the founders, of course, were really nice. Alex Povey and Brandon, they were just like, they didn't get mad or anything. I was contributing nights and weekends, kind of like I was at that other company. All of my keynotes were more like Kubernetes plus Coreos. And I remember at some point it was inevitable. We all got in a room. I became the product manager of Coreos at the time. And Alex was like, I think you got the vision here. And we got everybody in the basement in San Francisco, in the office we were in, like, hey, guys, all in on Kubernetes, Fleet deprecated. All these things that we're building deprecated. We're going to go all in. And I'm glad we did, because Alex Povey came up with the name Operators, which is like a core idea in the Kubernetes community. We put all this effort in there. Me and another guy worked on a thing called cni, which is the networking layer for Kubernetes. And what Kubernetes really meant for me was that previous 15 years of experience as a practitioner in the data center, learning promise theory, learning Puppet, where it works well, where it doesn't, and then understanding that Puppet wasn't the only way, and then making, going through all of these loops. And so when I ended at Kubernetes, it Felt like this would be the thing I would build if I only knew how. And that's the way I explained it to the rest of the world. And at that GopherCon, maybe the next one. There were people from the now early Kubernetes community. It was a small company called kismatic. They were kind of a Coreos competitor to some degree, but they came up with the idea, like, we should have a conference just like GopherCon. We called it Kubecon. Joseph got the logo going, and the Kismatic team kind of put up the money for the first event. And we really welcomed the entire community. And now it's been 12 years later, and the CNCF has done a fabulous job of keeping it going. Now there's like, what, 13,000 people here in Amsterdam keeping that thing going.
- [75:07]
Interviewer
So I asked this from Kat Cosgrove as well, who was on the podcast. What do you think really made Kubernetes breakthrough and then just become the de facto way of orchestrating nodes and just winning again? There was at Coreos, you were building fleet, Docker had Swarm. Like, as you said in the beginning, it didn't seem like this will be this big.
- [75:30]
Kelsey Hightower
I think the number one success criteria was Docker. So remember, there was Mesos and Mesosphere, and they had their own runtime. HashiCorp had come out with Nomad, and they had their own runtime. But the biggest runtime that had already got global consensus was Docker. So by that time, there were so many Docker containers and Docker workflows and Docker Swarm. Maybe the Achilles heel to Docker Swarm was its design. They tried to take the Docker API, which worked really well for one node, and expand it across multiple systems. And it was not the right API to scale to another type of thing that we needed. And so they kept trying. They tried to add storage, they tried to add networking, but the Docker API was never meant for that. And so the Kubernetes team was smart. Instead of trying to say, google's better than everyone and everything, they did a couple of things correct. Let's just use etcd. Let's just use Docker. So you take those two things and you take the experience of the people who wrote the Omega paper, which is kind of thinking about what would come after Borg and at least the things like Mesos.
- [76:36]
Interviewer
So Omega was a system where what Google would have built after Borg, but they never built it right?
- [76:41]
Kelsey Hightower
Yeah. So they had elements of it. So like the omelet, you know, this, like, agent that Would like be more declarative. A lot of hints from the Kubernetes world that will come later.
- [76:50]
Interviewer
And then just to be clear, Borg was and is still Google's way of managing their back then hundreds of thousands, now probably millions or tens of millions of servers. And they were best in the world with this or they say a lot. Right. So they learned a lot.
- [77:04]
Kelsey Hightower
I think Borg was one of these things where you integrate the hardware, the software, the package management, the configuration management, MapReduce. Right. Borg is this thing that just expands and grows. In some ways I guess it's extensible, but all of that insight and knowledge, but then they get so much experience with that. If you were to do it again, what would you do? And you read the Omega paper and it's like, here's how we, what we learn from scheduling. It doesn't need to be that complex, especially for certain workloads. You don't need this like high performance over engineered thing. There's a simpler way to do scheduling, especially if you can get the scheduler a bit more metadata about the workload. There are also big game changers now. Instead of talking about Java versus Python versus Ruby, you only have to talk about scheduling Docker containers. And so I think that's the number one success criteria that we were already off to a running start because you could just reuse the same Docker containers. You didn't have to rebuild a new image. Think so, given that what they tried to do in the early beginning was just fill in the gaps. And in many ways, yes, it's a new system, but it fills in the gaps. The one gap that they filled in was Docker had an entry point. So if you needed a Ruby app that needed NGINX and your process, you used to have to write a little shell script, the entry point script that would do all this magic, almost imitating an init system. Kubernetes is like, no, no, no, you don't need to do that. You can just make separate containers and then Kubernetes will run them as a process tree. And so for many people it's like finally, now we can have a clear way of thinking about application architecture. Like blocks, like blocks. Now instead of like you have to open the entry point to see what we're going to do versus full lifecycle management independent. So it solves that number one problem. The other big one that I think that they solved, number one, we went from infrastructure as code to infrastructure as data. And infrastructure as code is like if this do that bring in this module for loops, all this stuff. And Kubernetes is like, no, no, no, you have to specify exactly the containers you want, how much memory that they need, and then we have the status field to tell you if they were running or not. And you would take this data object that you could write by hand, give it to an API, and then the control loops would operate on this state. So that means it didn't matter if you had Ruby, Python or anything. You can just take your ide, write some YAML, give it to another tool, manipulate the YAML, and then pass it down to the API servers. You can build any combination that you want it without having to be a compiler first. That to me was a fundamental game changer that I don't know if a lot of people understood why it felt very easy to onboard to Kubernetes. Kubectl apply object, off you go. And the last thing I think credit to Brendan Burns, the ability to extend Kubernetes in a first class way. OpenStack didn't have it. Mesos didn't really have it. In Mesos you have a scheduler and you built the other part of the scheduler so you can have Spark, Hadoop Marathon, but you had all these other tools sitting on top of a thing. So an extension in Mesos was heavy, almost like a whole nother system. The thing that makes Kubernetes powerful, there's a data model. We gave infrastructure a type system. So instead of imperative shell scripts, you finally had types. So if anyone's ever come from like Python to a type language, types do a lot for you in terms of cognitive overhead. Like you really know what goes into this function. If you put the wrong thing into this function, it doesn't even work. Like you can't pass a string where integer needs to be. Kubernetes brings the same semantics to infrastructure. And finally, now it's much safer to automate things because you're gluing together things that actually have structure and types.
- [80:42]
Interviewer
I was about to say the reason we love types and every language goes towards them is safety and it eliminates a whole class of errors.
- [80:49]
Kelsey Hightower
Yeah, you could do things like static analysis. You can have other tools compile different things and ensure that they have the exact same thing. And then you have this validator that tells you that's not the right object, that's not the right field. And so once you have all of those things, you can build a really nice deployment system. So kubectl deployed these containers, no problem, but what about everything else? So instead of trying to evolve Kubernetes to do everything, Brendan Burns, I remember sitting next to him, he's like, kelsey, let me show you this thing. You can extend Kubernetes just by giving it a description of what you would like your object to be. So if you're thinking about this from the rest world, hey, I need a user, here's the crud operations and just give it to the thing and it does everything else for you. And so when that came out, it's like, so if I wanted to manage, let's say a firewall, like, yeah, you can describe a firewall and you give that to Kubernetes and all the tooling works, you can now say kubectl apply a firewall. And so now we got tools like cert management, where if you want a certificate from, let's encrypt, you can just say what domain you want, where it should live, give it to it. And now you had a first class extension, you didn't have to uninstall it. You have to make some magical binary. And it really didn't matter what language you wanted because once you put the data model in place, you got the machinery and if you care, if you like Python, you can have Python running a loop, grab the data and then make it so if bash was your thing, you can literally pull the config using a bash script and make it so and just update the status field. That opened up the entire ecosystem. So Cisco could come in, do what they wanted to do, Red Hat can come in, OpenShift and do what they wanted to do. To me, that was the game changer that brought the rest of the community in.
- [82:29]
Interviewer
And then you joined Google, not very surprisingly at this point, I guess from the outside, of course. But how did that go? You've been contributing to Kubernetes as well. The team was there. Did you join the Kubernetes team?
- [82:42]
Kelsey Hightower
No. So by that time I met Coreos, Kubernetes has definitely taken off. I'm giving lots of keynotes now. Everyone wants to know my opinion. I'm making all these prototypes, I'm kind of moving things forward. There's a Kubernetes book now, right? I'm a co author with Brendan Burns, Joe Beta at the time, I'm like, you know what I am, I'm thinking about my exit. So in our careers we do a lot of work to get into this field. All the certifications, the boot camps, the studying, some of us college. And then once we get in, we're thinking about career progression. Dev, senior engineer, principal engineer, distinguished engineer. And we spend almost our entire lives on that trajectory. And our field is so young that some of our pioneers are finally, like, no longer here for the first time. We're not used to that. And we're not really used to people retiring. Like Rob pike just retired. The concept of an exit for an individual contributor or a leader in the tech space is. We don't. We didn't have a lot of those. Linus is still at it. He's not retired. Right. So we don't spend a lot of time thinking about the exit plan in our field. If you're a professional athlete, your body will tell you when it's time to go. And so at Coreos, we reached this peak. I felt like I've done everything I've
- [84:02]
Interviewer
ever wanted to do in the tech
- [84:04]
Kelsey Hightower
industry, in the tech industry, from 1999 to being unsure of myself, to seeing myself on the side of museum buildings, full landscape view, because people are coming to see what I think about where technology is actually going. And so it comes full circle. You get a bit of taste of the fame. You can look at GitHub and you meet people. We use your libraries. We've used your command line tools. I started my career like, some people were not even born when I graduated high school, and they started their careers from those books. And so I felt at the time that I had come full circle and I was starting to think about the exit part of that journey. I remember spending time with Jet Propulsion Labs, jpl, part of NASA. And I remember being there, and I was so excited because the movie the Martian had just wrapped up filming there, and they gave me a tour of the facility. Like the Mars Rover, the new one, before they launched it to Mars, they were qa and it was just going in a circle around the track. And I'm like. Had a little laser on there so it can split rocks. And they were showing me how they improved the wheels over time. I went to another lab and there were a bunch of scientists working in there, and it looked like a fish tank. And I described it that way. It was like, yeah, we determined that if you want to replicate parts of Mars surface, and I might be getting this wrong, that the rocks that you find in, like a fish tank, we can replicate some of this stuff. And maybe it's slightly different than that. So I'm watching these people work, and he showed me, like, how spacecraft has evolved over the last 20 or 30 years. I'm like, wow, you all seem to have an actual purpose. For the first time I've seen people using technology not to just make more apps, not to add numbers in a database, but to actually have humanity do something. And so they were not all about Kubernetes and Docker and Python and go. They were like, we're just trying to get a person to Mars and back again. I remember a part of the interview process. Their interview questions are, if you had to deflect a meteor, how would you do it? But it has to hit one state. So now you're in the leadership position, what would you do? And you're just explaining the answer like, you know, I would kind of bring a bunch of experts and then, you know, you got to think about the ability to evacuate people and you have to explain yourself. And the core part of my answer was, you would have to explain yourself. Almost 24 hour live stream. Here's the trajectory, giving everyone the countdown, explaining every decision you're making. I chose transparency. I chose truth. I chose like, look, we have to deflect it. There's no way to make it zero. So we've chosen this state and this is the evacuation plan and we estimate that this number of people won't make it. We're just being honest with you. No need for conspiracy theories. It's live. And I was like, wow, what an interview. A 20 year career at that point, never had an interview that made me feel that way. And so I actually was going to go to NASA after Coreos. I even signed the employment agreement. I was going to move to Pasadena, California and work at NASA on the Mars mission and lead up the infrastructure and the infrastructure teams. And of course Google called and was like, hey, come to Google. And at that point I was like, for what? You have hundreds of thousands of employees. I admire Google. I've been there before, but not in that capacity. I was gonna go to the headquarters.
- [87:20]
Interviewer
Yeah. Not to the, not to the data
- [87:23]
Kelsey Hightower
center, but to the headquarters. And I've always admired people like Brian Grant, Eric Tune, Don Chin, all of these wonderful people that I got to work with through the Kubernetes community. In many ways I felt like I was already working on the team because by that time I had commit access to Kubernetes. So I kind of felt like all the things I wanted to achieve in that regard was there. And so I was like, why would I come work there? I'm just going to be a cog in the wheel. I'm going to go there, I'm just going to disappear. They're going to just make me work on Google stuff all day long. What's the value in that? I've seen the peak of this. And they were like, we won't do that. I was given the opportunity to do Devrel and it's the first time I ever did it. But Devrel represented freedom. No tickets, no write code, measured against Sui benchmarks. I was like, I don't want that. I wanted to be able to make impact. And so the team was smart and they were like, look, we got this area called Devrel and we'll let you define what you do.
- [88:16]
Interviewer
So you got to write your job description.
- [88:18]
Kelsey Hightower
Pretty much, yeah. But as an entrepreneur, I know how this goes.
- [88:22]
Interviewer
How does it go?
- [88:23]
Kelsey Hightower
If you come in and you really do Devrel stuff, in my mind, you're going to get fired. Because if you limit yourself to the external perception of Devrel, you go to conferences and you become more of an evangelist. You do tutorials and guides. For me, those are activities. I'm person of impact. And so the first thing I did when I got to Google was like, where's the customers? How do we make money on cloud? So I need to figure out how to talk to the customer. So, hey, where's the sales reps? Hey, if you ever need anyone with Kubernetes expertise, call me. I will fly to Disney, I will go to Walmart on site and I will whiteboard for six hours because that's the revenue component. So globally. Went to Australia, Canada, doesn't matter. We don't even have a region there yet. I'm going to bootstrap it. So now I'm growing my impact on the revenue side.
- [89:11]
Interviewer
And the reason you said if you would have gone as default, you would have been fired because you would have not been generating any revenue.
- [89:16]
Kelsey Hightower
I just felt like I was going to be fired.
- [89:19]
Interviewer
But. But this is just like thinking as an entrepreneur as like, you want to make money.
- [89:23]
Kelsey Hightower
I want to. I want to make sure that I'm impacting the business. And for most businesses, revenue is the criteria. And no one ever made me do that, by the way. It wasn't like, Devrel, we're going to hit you with the metric. It's like, no, no, no to me. I understood the value of revenue, so I would go out and I was able to do it in an authentic way. I'm just talking about the same things I was talking about before. And then product impact. This is cloud. Why limit yourself to Kubernetes? There are serverless, there's databases, there's metrics, there's so many things here. So now I'm like, I need to learn everything and I want to employ all of my skills. So it turns out my time in financial services means I could be an exec sponsor. I knew how to go from hello world to hello revenue. So if I got into an exec briefing, I didn't waste everyone's time showing them the latest feature of Kubernetes. Doesn't make sense. They want to know how these tools come together to lead to actual impact and outcomes. And so I matured there and also got smart. You got to go to other teams and you read the okrs. So another team might say, hey, Kelsey, we're really trying to get more adoption on our metric stack. And I remember the first thing that I started implementing at Google was a thing called empathetic engineering. So you have a lot of smart Googlers. These people are brilliant. I mean, extremely brilliant, to the point where the hardest problem Google had, in my opinion, was what to build, not how to build it, what to build.
- [90:45]
Interviewer
They could do that.
- [90:46]
Kelsey Hightower
And in some cases, you end up with like, five messaging systems. But the thing is, what to build seemed to be the most pressing problem. And so the one thing that I tried to do was like, how do you convince other engineers, you know, their manager, how do you get them to trust you? And I started these empathetic engineering sessions where the first one was like, get the Kubernetes team in one room, all of them engineering offsite. I want you all to install Kubernetes, but you can't use any scripts. And remember these. Some of them are distinguished engineers and principal engineers. Some of them worked on Borg. Some of them are just the original creators of Kubernetes itself. And it was so fun to watch them struggle, because it's like, do we install Docker first? What version of Docker can we put this on Ubuntu, or does it need to be red hat? And so an hour goes by. Teams of four are like, nah, man, this doesn't work. I was like, great, you all can stop. I'm going to show you how I would do it. And of course I know how to do this, because I get to prepare, right? So not a knock against them. I'm just like, all right, Debian, tune the kernel this way. Put Docker on there, put Etcd, put the API server, put all these things. There you go. That's how you do it. And I was like, yeah. I mean, of course you had prepare, of course. And so the question then was from an engineering perspective, how will we make this better? And then people were like, well if we had OS packages, this could have been advocate install and we could have just used local machine. It was like, that's a good idea. And someone was like, we will make that happen. Another person's like, even if you have those packages though, you still need to know what order and where the config files go. So Kubeadmin was born, which was a command line tool that gave you a procedural thing. But the other thing that I remember from my career was I don't want just a tool that abstracts everything from me. I want to know how it works. So I wrote the guide Kubernetes the hard way. And that guide is what I use to help teach people at GitHub in the early days pre Microsoft how to run Kubernetes on bare metal and walk them through that guide. And so it was that empathetic engineering that helped me make a huge impact on cloud. Because I can go to every team, every org, and instead of guessing what their roadmap should be, given someone who would spend time in the field, given someone that had this enterprise background, hands on experience across lots of tooling. I knew where people were coming from. I would say based on where Google sits in the landscape and in the competitive landscape, given what our abilities are and what our customers need, I think this is it. But I never said it that way. I would get everyone in the room, you would have them discover it, have
- [93:11]
Interviewer
them discover it and nudge them. So you knew what you think the key, the most important problem areas were. For example, and then you orchestrated a session to help people try to always
- [93:25]
Kelsey Hightower
know the two things in multiple product areas that would make the most impact, AKA revenue, that would get adoption. And so, and then launches versus landings, you would put all these things in motion. They would land at different times. So launches is we shift the thing. People have a big celebration. I know that doesn't matter as much as the landing. People are actually paying for it. So when I got good at that cycle, the promotions were a little predictable because I would be able to make impact. Sometimes you built prototypes and sometimes you would contribute to things like cloud functions. But the overall goal though was to use every skill you had. So working with PMs, we got to add GO support to cloud functions. We're the team that made go. Amazon has support in Lambda for go. We need to do this. And also I have a keynote at the next Gopher Con and I'm talking about serverless and I Have two choices. I have to use Lambda or we can use Go functions. And so we sprint, we get it done, we get all the Googlers reality, we get it checked in, it's in alpha. And I tell the PM, hey, you want to go to GopherCon? We'll launch it on stage. We launch it on stage, we tell people how we designed our worker, how the internals work, and then they can just come sign up. That was the trifecta. You do the work, you have the vision, you execute, you launch and you land. Land and the landings. Compiled over seven years is how you go from like I think maybe L5 to distinguished engineer L9.
- [94:53]
Interviewer
That's like four promotions.
- [94:55]
Kelsey Hightower
Yeah, four promotions, seven years.
- [94:56]
Interviewer
But it's interesting because a lot of times when you ask someone on, you know, how they got promoted, let's say, yeah, four, four times in seven years at a place like Google, I would expect, just being naive, that they will tell you like, okay, this is how I planned or like this is what you need for each level. But sounds like it's a very different. You just had landings that created impact and you were focusing. Do I read it correctly that you were not focusing on your promotions or the next level, you just wanted to do the best work that you could?
- [95:25]
Kelsey Hightower
Oh no, no, I was focused on the next levels because that was the goal. Because think about it, if my whole career I've always tried to acquire the skills and make the impact so I can move to the next level. And so at Google the levels were expressed as promotions. And there was a point in time in the org that I was in, there was no level 7 for an individual contributor. So now we have to make a level seven. And then once you get to a level seven now you kind of pave the way for the other people that want to come up through that IC path. And then level 8 isn't the same as level 7, level 8 isn't the same, or level 9 isn't the same as level 8. And yes, there is a formula to some of the promotions early. So from 3 to 4, 4 to 5 is a little formulaic. You have a ladder, there are things that are expected of you and the decision making on those type of promotions are localized, meaning your manager, maybe a director. But then outside of that though, when you start to go higher, it's now expanded where there's now other teams that have to understand your impact in a way that can't be biased by a local team. So if you think you've made a lot of impact and then people across the Org do not agree. That allows you to really throttle what impact means. Right. Because if it's just your team, I really like this person. Now everyone's a distinguished engineer. And so I think Google did a really good job of saying, look, it was okay to be like L5 or 6 for your entire career.
- [96:46]
Interviewer
Yeah, that's their terminal level. I think it used to be L5 and now it's L4. Actually, they moved it back because I think L5 has gotten a bit tricky and they don't want to fire really good L4s.
- [96:57]
Kelsey Hightower
Exactly. So. And I think for a lot of people, you can follow the formula and get to where you need to be. But I was like, at the time, I think there was like, a couple hundred distinguished engineers. So as an entrepreneur, if you show me the top of the mountain, I want to get there.
- [97:10]
Interviewer
So then you were targeting. You were like, all right, how can I get to the next one? To the next one. And impact was the name of the game, right?
- [97:17]
Kelsey Hightower
Yeah. And I guess the only thing I would probably add to the moment or to the situation was I wanted to do it authentically. And so there was one part of the time I remember I didn't get one of the promotions, but I was getting promoted pretty fast. So of course people were like, dude, you need to actually make an impact before you get promoted again. Stop it. And that was good feedback. But I remember I took a chance. So there's a pattern to doing a promotion packet structure. There's examples, and you go through all of this stuff and you get feedback. I remember one year I was like, I'm not doing that this year. I'm just going to talk in, like, the first person. Hey, I'm Kelsey. I work on these things. Not these things. These things are important to Google. So here's exactly what I did. But more importantly, here's the people I brought along, here's the teams I've impacted, and here's the results of this. So I did this project, and here are the results of that. And I'm talking in this way like I'm ignoring the process. I don't really care about the template. And as someone who at that point in time was on some of the promotion communities where we're looking at the promotion package and making a decision as a team, I decided to write my packet for those people so that when they got it. Because, look, if you're having to do a read a lot of these, it's hard. You're like, oh, man, they all are so dry. Everyone's being very safe. Everyone's only telling me what I want to hear. I don't even know the person from. After reading this whole packet, I said, I'm not going to do that. I want them to see me as if I was in the room advocating for myself. And I remember getting feedback on that packet like, kelsey, this is like, come on, bro, this is, you know, things. I was taking it seriously, wasn't. I wasn't making a mockery of it. I just wanted to make sure that they understood what I was doing. And I was aware that I wasn't just trying to play the game. I was trying to approach this process authentically. And I got promoted off of that packet. And look, it could just be sometimes the work sometimes speaks for itself, but a lot of times people can't see the work if it's not presented correctly. And so that kind of slingshot at me and the reason why I tell this story is that every distinguished engineer doesn't get there the same way everyone thinks, oh, how much code did you write? What complex thing did you build? And for me, I think it was the impact on Google Cloud's culture. The empathetic engineering thing became an official thing that they used to onboard other engineers. It became things that had a whole team behind it. They used to go give them and a product manager that would evolve the program and integrate into hr. The philosophy around other engineers saying, kelsey, we just did an empathy session. I want to show you the results of it because we're about to ship it. And then engineers started really thinking about the customer. I mean, Amazon was always known for that to some degree. But to help Google get on the same page, I mean, I'm pretty sure other people had an impact, but there was a direct line of impact from those type of programs. And also me diversifying, moving away from Kubernetes into the serverless realm, moving to the world where you're helping out the postgres or the Spanner team at a postgres interface, the Go team getting a little bit better with Cloud, just making other people successful around you is one of those things that helps you become distinguished. It's the impact, the ability to influence.
- [100:19]
Interviewer
Before you got to the distinguished level, you shared a story about Microsoft and an offer. Can you tell us about that?
- [100:25]
Kelsey Hightower
Yeah. So I'm one of those people. I don't like ultimatums. It's hard for an ultimatum would be you're working at a company for a couple years, you're doing a really good job. I'm just going to make up numbers here just to protect everybody. But let's just say you're making $100,000 and for a lot of times you could be happy with 100,000. And so a company says, hey, we're hiring. And you go there, you do an interview and they say, look, we're paying $120,000. At that moment you are actually worth $120,000 because you now have evidence from the market. The 100,000 you currently make doesn't look so good anymore. You can't unsee a job offer for 120. And so now you have to make a decision. You can commit to that, leave your job and go make 120 and start over, or you can do the ultimatum thing. If you don't pay me 120, then I'm leaving. And that puts everybody in a weird predicament because sometimes it doesn't have to be that adversarial. Sometimes it's just this is evidence that, hey, look, I want to advocate for myself. I know we're not in promotion cycle, but I believe I'm worth 120. And I would like to have that conversation. And someone will say, hey, well, you need to go prove it. It's like, well, I have an offer if you want to see it, but I would rather be here. And that can turn into a you went to go look for another job. It's like, oh my God, so I'm stuck here.
- [101:46]
Interviewer
It can have weird dynamics, right? Especially with your manager or your management chain.
- [101:49]
Kelsey Hightower
Exactly. So I was always wary of that. But I also knew that in business that's just how it goes. And so I'm really at the peak now. I'm thinking like early retirement. I'm probably can get out of here at what, 55, 60 if I continue on this pace. And then Microsoft was like, hey, come through. And look, I never had an executive recruitment process. So by this time I'm probably considered an executive. At Google, once you hit like L7, you're kind of considered like an executive.
- [102:19]
Interviewer
What is an executive process?
- [102:20]
Kelsey Hightower
So an executive just means that you probably have an admin to help you with some of the tasks you're doing. You're probably going to be asked to be an executive sponsor of things and programs. So if a team wants to have a program for something and there's going to be a budget for that, you may have to help oversee that program. So they need executive sponsorship. Another engineering team knows that they're going to need budget for something. And they need someone that has a little bit of political capital, a little bit of weight in the organization to help endorse them. So that can be executive sponsorship, or you might be assigned to one of the largest customers the company has. But that executive set of duties means that you're going to be making impact above and beyond yourself, typically to support other parts of the organization. So I grew into that at Google. Microsoft was like, we want you to start there. And here's the thing. I didn't think about my role that way there. I'm still the old Kelsey from coreos days, right? I'm just doing well here. And I wasn't going to interview at Microsoft. I'm like, for what? I'm at Google. I mean, I had a lot of freedom at Google. I was making impact at Google. I had a good reputation. And so I'm like, no, my trajectory is fine. I'm not doing that. And to be honest, I don't like Windows. I didn't like Azure. I don't like. Net, I like GitHub. VS code is nice. But my whole career, the majority of it, has been rooted in authenticity. I've been working on the things I actually like and care about. This would be one of the first times, maybe outside of some of the enterprise roles, that I'm going to go work on a set of technology that I wouldn't use on purpose. And so I went there and I met the Microsoft Team. And the weird thing though is I had a recruiter and they swapped the recruiter out and it's like, hey, our mistake. I'm like, what do you mean mistakes? Person is super nice. And they asked for a resume and I didn't have a resume because it's been seven years since you had to create one. Well, it's been actually 15 years at that point since I ever had to show someone a resume. So I'm like, resume? Oh, man, I haven't made one of those in a long time. I'm looking online for a template. Like, what's the style of resumes these days? And so I'm like, trying to figure out a resume. I'm like, see, this is why I don't waste time interviewing. I don't got time for this anymore. And they swapped the recruiter out and it's like, hey, sorry about all that. We don't need no resume. Sorry, that's not this kind of process. What we want to do is make sure that you meet the right people who represent Microsoft. I was like, okay, that's different. So I get on site, I'm like, okay, what kind of quiz questions are we going to be prepared for? Because I'm not doing quiz, bro. You had commit access to Kubernetes. You wrote the book on Kubernetes. You're leading these things. You helped start Kubeca. Come on. Like, you have a Wikipedia page. We don't need to try to figure out whether you make the impact. We can just look at GitHub. There's evidence there. And so I'm meeting all of these leaders. They're bringing all these people who are behind the scenes. I remember meeting Scott Gunthery for the first time, and I didn't really know who he was because I just didn't know how. I didn't know the lineage and the history of Microsoft. And, you know, of course I'm looking up who these people are. And I was like, wow, this is. They're courting me. Oh, this is what that feels like. I mean, Google did it a little bit too, but not like this was like, I'm looking at the. These are the executive directors of.
- [105:31]
Interviewer
This is like reverse interview. They're not interviewing you. They're trying to sell themselves.
- [105:36]
Kelsey Hightower
You know what? And that's the thing that I appreciated a lot because they were like, this is your career. You've built a fantastic career over there. We can't ask you to throw that away without understanding what you would be doing here. And so it was like, you can, if you want to understand the business, if there's someone you haven't gotten a chance to talk to, just give us a name. And I'm talking to them. And the one thing that I really respected about Microsoft was I think by that time they had also acquired GitHub. And so they had a big vision for themselves, a lot of diversity. And I was like, okay, there's a lot of opportunity here. They're also all in on Kubernetes. They had just acquired people like deis and Brendan Burns is there. So I was like, all right, Kelsey, you can come and make an impact. There's room to grow. I'm like, all right, I might do it. All right. So I'm glad I did the interview and I get home and it felt like that time I doubled my salary. I told you on the way home. And I remember I get this email from Satya, the CEO of Microsoft. I'm like, man, he wrote this nice email. Kelsey, I heard you had a good experience with the team. Remember, I did the interview at the Microsoft headquarters, right? So hey. Heard really good things from the team. Just wanted to let you know, you know, you're going to be respected here. We're going to support you as a team. I'm like, damn, support as a team. Coming from the CEO. And the offer was like a PDF. It's an attachment. So I read this thing and so, number one, what an honor. This is the CEO of Microsoft. He has so many more important things to be doing than to be emailing me about a role. And I open a PDF. And very often in your career does a zero get added to the equation. And so you're looking at this like, I didn't even know that they do that. We know that it happens, but the person that graduated from high school in 1999 that chose a certification didn't know that was available even while I was at Google having all this success. And Google paid me pretty well, too. But I know you can add another zero still. And so I'm like, whoa, this is. This is crazy. And I'm like, wow. So I. I showed my wife and she was the one that said, you should just go interview. Like, put your ego to the side and let's go see what's out there. So shout out to my wife. And so I get the PDF and I'm like, okay, this number is perfect. Honestly, I don't know what to say. But just, just find out, like, is this really the only number? So I remember giving a counter. Like, you know what? I think it should be this. And the funny thing is Microsoft counter back higher so we're not playing around. I'm like, oh, now, now I understand that. I don't understand this part of the game.
- [108:07]
Interviewer
Yep.
- [108:08]
Kelsey Hightower
And so I have this offer, and I knew that I wasn't going to go interview there if I wasn't serious about taking it. So I was serious about going to Microsoft. Been almost six years at that time at Google. And I went to my manager. I had the same manager, which is legendary at Google. Had the same manager for six years straight, even with all the reorgs. Same manager the whole time he was a director. So even when I got leveled up, they allow at least someone one level above still to report to a director. And I had such a good relationship with them. I told him what happened. I said I wasn't looking, and they asked me to come in. So I went in and I'm going to take it. Number one, it will be financially irresponsible to not do this. So that will be the driving force. And also I get to Stretch myself in another way and see if I can make an impact again. He's like, okay. But it was no ultimatum. I was like, I'm leaving. And he's like, I'm just curious what Google would say. I said, no, not great. I don't want to do ultimatum.
- [109:09]
Interviewer
Yeah, you don't believe in this.
- [109:10]
Kelsey Hightower
I mean, I do believe in it, but I didn't want to do it.
- [109:12]
Interviewer
You don't want to do it?
- [109:13]
Kelsey Hightower
Yes, I didn't want to do it
- [109:14]
Interviewer
because you understand the dynamics.
- [109:15]
Kelsey Hightower
The dynamics, especially I. In many ways, Google had been really, really good to me in every facet. So it wasn't ultimatum time. It was more like, I've earned it. And he said that too. I gave him the PDF and he looked at it. He's just started smiling and like, oh, wow. Whoa. And then he said something that was really dope. He says, I want you to know you're worth every penny of this. And I was like, oh. The whole.
- [109:42]
Interviewer
That's.
- [109:43]
Kelsey Hightower
That was a game changer because he knows me like no other person I've ever worked with. No one knows me as well as Greg does. He knows my strengths, he knows my weaknesses, he knows my ambitions and my motivations. He knows all of them. For him to say that was important. And so I'm thinking if Google wanted me to be at that level, they would have done it already. But also, pragmatically speaking, no one really knows your situation. Like, all these people have thousands of people. They manage. They're not targeting you or being mean to you, at least in my case. And so maybe they thought I was making that kind of money already. Maybe they thought that I was already a distinguished engineer. How would everyone in the Org know? They got other things to worry about. So he presents the thing to them, and I think within hours, they're like, this is no problem. Like, hey, here's the. You know, don't worry about that number. Here's a bunch of stock and all these things, and I'm looking like, whoa, whoa. So now I have the money and I'm at the company I want to be.
- [110:44]
Interviewer
And you didn't do an ultimatum?
- [110:46]
Kelsey Hightower
I didn't do the ultimatum. So I felt good about the relationship. I didn't feel like there was going to be some retaliation. I had no fear about that. And I continued to be successful. Eventually got promoted to distinguished engineer at Google. Told the Microsoft Team no. But the one thing that people saw from this, I didn't talk about this at that time, but I did do a TWEET and the tweet was different company, same team. A lot of people was like, what does that even mean? Different company, same team. But people were retweeting it and they liked it. And people just thought, oh, it's because of the community stuff and how the different people in the community work together. And that was part of it. But a big part of it was that moment that Microsoft got me the biggest raise in my career at Google. And about three to six months later, I was in San Jose and Satya was there. And his admin is like, hey, Satya would like to just, you know, meet you, with you. I'm like, the CEO of Microsoft. Got time to do anything. So I'm in San Jose and then I go to this hotel and the admin meets me downstairs. Like, hey, Satya is kind of ready to see you now. And I'm going up. And now he know I'm coming, right? Like, let's just say the meeting's at 1 o'. Clock. He knows I'm coming for, like, days. And then you go into the hotel room and, like, the. The doors are open, overlooking a mountain range, and Satya is sitting there overlooking the mountain range, like a Vanity Fair photo shoot. And I remember before on. On the plane ride there, I'm reading his book, Hit Refresh, and I remember the opening chapter. He talks about starting Microsoft as a developer advocate and now being CEO of Microsoft, advocating for the soul of Microsoft because he had been there so long, he saw the birth of the cloud and all these things. I was like, oh. And the book was actually a pretty good read about his trajectory in the industry and his time at Microsoft. So now I'm done with the book and I'm about to meet him. So I have all of this context in my head. So when I walked through, I felt like I had this rapport with him for some reason, because I read the book and then he's looking out over the mountain range, and I walked through the door. I was like, sati, why are you sitting here like this? You knew I was coming. So you're just posing like we're doing a photo shoot. And he just started laughing. And so the tension, at least for me, was way down. And we had a discussion. I won't repeat everything here. But he said something dope. He said, we were sitting around at the table and we asked ourselves, what executive did we want? That got away. That means that you still were in the minds of at least some people. And it's like you were on that list and we had a discussion and he was very transparent and he said, we gave you a good offer. We think we gave you a good offer. And at that time, Thomas Kerry had just come from Oracle. My personal opinion, I liked his leadership at Google, but I can understand why some people were afraid of the Oracle DNA being brought to Google. And I think maybe some people in the industry felt like, oh, this is a moment to go and maybe poach a few people that didn't want to make that transition. And he said something like, we gave you an offer, as if you were running away from something and we should have gave you something to run towards. And I was like, damn, that's poetic. And so we wrapped up that meeting and I really felt like, man, I actually belong to the industry. It wasn't like I'm a Googler or a Kubernetes person. I really felt after that moment that I was an industry person. So I was very comfortable at that point. Retiring within the next year or so.
- [114:14]
Interviewer
You talk with Satya Nadell and about a year later you retired. How did that thinking come up to?
- [114:22]
Host
Because you were mentioning that you kind
- [114:25]
Interviewer
of had your retirement. You were thinking about your kind of end game or exit game.
- [114:29]
Kelsey Hightower
So before joining Google, I kind of felt it was going to be possible, right, because now I'm making money, I'm saving. I really practiced a whole life of minimalism. You know, I live way below my means. So even when the money changed, the lifestyle did it. I was very conservative in terms of what I was spending. I didn't care about jewelry, I didn't care about buying cars to impress other people. That was gone. I felt free from that kind of thing in society. So the money started to become like freedom tokens. This money means I can get out of the game. And so everyone, at least the people I know or what I attempt to do, you set this number and then when you blow past the number and if you're still a little young, you're like, well, maybe I can change the number. But the thing I was careful not to do was to change the lifestyle because I watch people around me change the lifestyle so you can make a lot of money and still be broke. And I was like, I want to avoid that. Because at that point I used to ask questions, why am I doing this? And for me personally, I felt like I was lying to myself. I love this job. And it's like, no, you learn to love it. And being someone who was good at it, you tried to figure out working around the people you like. Helps you love it a little more. Working on the things that you're curious in helps you love it a little more. But you can't deny the pressures of just enterprise stock price. You know, this person wants it. Now there's a debate on whether we should do this or do that. There's personalities involved. There's parts of it you didn't love. And then the other one I thought about was time. Everyone thinks they're gonna probably live to 90 or 100 or we don't even think about it at all.
- [116:07]
Interviewer
I think a lot of us just don't think about it.
- [116:10]
Kelsey Hightower
Right. And maybe it's not healthy to think about it, but at some point, I think around maybe 37ish, I'm thinking like, what's the point of doing all of this work? Why are we doing this? And I used to ask that question in my job, why am I doing this? Why am I writing Python? Well, it's because you're a software developer. It's like, that's not the answer. That's the easy, obvious answer. So I learned to just zoom out. And when I started zooming out on my career, it's like, what are you doing this for? Because once I started having a better answer when my daughter was born, I'm doing this for her, I'm doing this for my family. I want to make sure that we're all safe and protected. And so I started just changing my attitude. So, for example, my daughter was born. I remember just taking a job where I could work overnight. I'm just going to go work in a knock. I don't really care about this job as much in terms of career progression, as long as I can be home with her so we don't have to do daycare. I'll take my shift, my wife takes her shift. But this was the priority. So then I started to structure my work life around this. Now, I was never great at work. Life balance, I won't lie. My daughter goes to sleep, I'm back on that computer trying to learn new skills. I probably approached a lot of burnout in my career.
- [117:22]
Interviewer
I'm glad it's not just you.
- [117:23]
Kelsey Hightower
No. Yeah. I ain't gonna lie. I've been saying, like, oh, and I had a. I did not have that part figured out. And also the thing about burnout, what I think is interesting is if you play professional soccer and you put in a lot of effort and you lose every year, you're going to feel burnout. I'm doing a lot of work and we never Win. But the teams who actually win play more games than everyone else because they have to play the playoffs, semifinals. They got to win World Cup. And if you win it multiple times in a row, you play way more games than everyone else. But for some reason, the champions aren't tired.
- [117:54]
Interviewer
Yeah, they're not burnt out, because, I
- [117:56]
Kelsey Hightower
mean, they probably are. But they ignore the feeling because they know what's on the other side. And so my career had a lot of winnings. So as I'm pushing the limits, you're getting the win, you're pushing the limits. So some of the burnout that is psychological, you just dial it back because, like, wow, this was worth it. And so what the money became for me is that feedback loop of saying, you know what? Let me store these things away. Because here's what the math means. And I always calculated the math on interest payments, not stock increases. At some point, you get away from that. Like, I want vanilla US treasury bond. What does that pay? So how much money do you need to live off a fraction of this money? And I started just making the calculations. And then you started negotiating salary around these particular things. And you start asking yourself, am I making an impact that is worthy and deserving of these things? So then, of course, you get into things like investments, startup stuff, blah, blah, blah. But then I'm starting to say, oh, I'm getting close. I can see it now.
- [118:53]
Interviewer
Which looked impossible earlier.
- [118:54]
Kelsey Hightower
Yeah, it looked impossible. So, like, halfway in my career looked impossible. Maybe my 401k would do something. And how much does Social Security pay? And if I do everything just right. And then it was like, oh, I don't need Social Security anymore. I don't care what the stock market is doing, but I have to stay the course and be disciplined. And I have to structure my career in a way. So once I got that light at the end of the tunnel, I started making decisions based on that. I'm going to bet to go to Coreos instead of something that pays way more money. Ah, NASA looked great, but, man, this changes the trajectory over here at Google Plus. I'm going to have to step way up to be able to walk on that particular stage. At least that's what I told myself. But also, Google was the type of place that could pay for performance at that level. So I was like, wow, this is a good opportunity. And so now that I spent all these years thinking about, why do you work? I never had a good answer, but I never accepted the lie. And so I was like, I'm working to be Me and you become a distinguished engineer. But you realize you're a junior person. You didn't put as much work on learning how to live and the relationships and the things that you do when the computer gets turned off. You didn't put any effort into that. Not a lot. Not as much effort as you put into the work.
- [120:13]
Interviewer
What parts are you talking about? Is this the kind of the friendships? Is this the outside of work? Is this a community? What?
- [120:19]
Kelsey Hightower
Just developing. So most of my friends I've known for 20 years, we talk on the phone all the time. I see them, I fly, I see them. That stuff is important. I've been married for 20 years. I'm going to celebrate my 20th year anniversary in a couple of months. And I felt like the core parts of my life that I wanted to be healthy and stable. I think I did those things. But there was things where, you know, I remember going to Budapest for the first time and stayed an extra day. So it wasn't leave as soon as the conference is over. It was stay one more day. And I remember hanging out with people like Liz Rice and her team, and they were like, we're going to a bath. I'm like, what's a bath? It's like, oh, it's like big swimming pool. And we went to one of these huge parks where there's like, you walk in circle and then you go in the building and there's all these plunges.
- [121:07]
Interviewer
There's a station in your bath. Yep.
- [121:09]
Kelsey Hightower
And I was like, ah, this is not the kind of thing I do. The truth was, it was the kind of thing I never did. Didn't even know. So I said, sure, I'll go. So I went there. We rented some shorts because I didn't bring any swim things wasn't on the agenda. We were there for like three plus hours. And I was like, man, what an experience. And so when I got back home, I didn't talk about the conference, I didn't talk about the keynote. I talked about the escape room we went to and the three hours we spent at the bathhouse.
- [121:40]
Interviewer
The escape rooms are also good.
- [121:41]
Kelsey Hightower
Yeah. And so I was like, what the hell was I doing? I was moving too fast through this thing. And so then I started dialing back a little bit. So, hey, I gotta dial back a little bit. And then little things like, as a minimalist, I always tried to live intentional. So it wasn't like the first time. I just realized there were other areas where I wasn't being as intentional. I remember when I was listening to Music. I was like, hey, my wife can sing and she knows the lyrics. So sometimes I'm singing a song, she's like, that's not how that song goes. And like what I remember. And now when I listen to music, I actually pull up the lyrics and I listen while reading the lyrics so I can really understand what the song is about. And it's little nuanced things like that where I was like, hey, life doesn't need to be so fast. And this is why you see me sometimes online. Like the fact that people are over indexing on productivity doesn't necessarily sit well with me because it's like if you just do productivity, you're gonna miss everything. You're gonna miss the experience, you're gonna miss this part that are hard. You're gonna miss the collaboration with your team. If you just go through too fast, you're gonna move right past it. Because you're a human, you're not a computer, you're an actual human. And you don't work only for productivity. Maybe that's what your job believes you are. And there's that saying you're not your HR title. And so for me, part of that was like dialing back to like, oh, you're a human, act like one. So I started to invest in like my relationships, talking to people, being patient, go to the school board meeting, do some of the off site things, spend more time with your child in her games, teach her how to drive instead of going only to driving school. Cleaning became a big part of my everyday routine. And look, if you have enough money, you can hire someone to clean your house for you, no problem. And there's actually nothing wrong with that. But boy, I enjoyed the parts where it reminded me of all the success you've made. There are some people who don't have a stove, there's some people who don't have a refrigerator. And so when you would clean them thoroughly, like take everything out, look at all the expired food, sort everything back out, clean it back to the condition when you bought it, put it all back together. And what I noticed was for the people who had, in my opinion, success, they could afford to do that. It is a luxury to be able to afford to go slow because when you're really, really busy, you need an admin because you can't afford to book your own tickets. When you're really busy, you cannot afford to clean your own house. And some people would say, I make way more money having someone else clean my house and I can go make money doing everything it's like, I promise you, money isn't everything. It is not. And so for me, I said, wow, now I have time, because some people only have money now I had both. And I decided that I could actually slow down. And the thing that maybe some people noticed, even my outward projection changed. I was way more methodical. And so the work changed, the keynotes changed, because I started to incorporate the philosophy. I brought the people into the keynote and I started asking questions like, I know what I want to show them, but then how do I want them to feel? Because I wanted. Sometimes I wanted people to feel excited, sometimes I wanted people to feel a little bit embarrassed by the state of our industry and the complexity that we added for no reason. And I noticed I was just starting to be like, the full Kelsey was starting to be on display. So then I was like, all right, it's time to retire. What am I going to walk out to? And luckily I was practicing just enough of, like, who Kelsey is to start doubling down on that as a retired. And I didn't necessarily do a good job. I'm only three years into this, so I'm a junior retired person and I make time for lots of things, but I still want to hold on to all of this knowledge, all these parts of our craft. So I'm still doing advisory, I'm doing investing, I'm still doing public speaking. I don't speak as much about low level technology things. I do try to put a little bit of philosophy in there, but I know that I'm still holding onto that part of my ego.
- [125:36]
Interviewer
Can you tell me a little bit about the advisory and the investing, both, how you got started, what advising means? I think a lot of us software engineers are curious about this and some of us will have opportunity. And also the investing part. The good, the bad and the ugly. You can now speak freely.
- [125:53]
Kelsey Hightower
Yeah. So look, I think if you're a software engineer, as you progress in your career, you will be an advisor. Because if someone wants to build something, a junior engineer does exactly what they ask. If I get this ticket, I want to do a good job. I want to get it done on time exactly as you ask, bugs and all. And then as you get more experience, you know that the person asking may not know what they're asking for. So you're going to be an advisor. Hey, if we did that, it's going to add a lot of complexity and you're not going to get what you want. What I think you want is this. Let me show you. And as an advisor, you're not necessarily the person's boss. You're just trying to give them something that they don't have. So you're advising. Sometimes you become an engineer and you get really close to the executive team where they check with you before they make any big decisions. So you're an advisor. When you start to advise at a very high level, then you also share in the outcomes. Right. A lot of software engineers that work at larger tech companies, they have equity. So if you just don't focus on heads down, do your job, and you get into more of those advisory roles, then you start to realize that maybe you start to have a little bit of effect on the stock price. Right. So my actions, if they turn out well, then we get there. So, okay, so how does advisory work in the startup world? When I first started doing feels like an exact waste of time. Number one, VCs have large amounts of money, so they can invest in a thousand companies, and ideally one of them will return the entire fund. But when you and I do, like in angel investing or advisory, typically when I started advising, you would take advisory shares 99.99% of the time. They're worth absolutely nothing. Number one, you're going to get diluted to hell and back. Oh, yeah. You don't even know how the taxes work on this thing. You may exercise them in the wrong way and you may end up paying money.
- [127:42]
Interviewer
Yep.
- [127:43]
Kelsey Hightower
And getting nothing in the end. So then you start having this allergic reaction to, like, advisory shares, especially if you get the ones that are way low on the totem pole. So then I'm like, you know what? I'm never going to work for free. I used to say this to myself, you can't be working for free. Can't be working for free. And the people who respect you, they'll never let you work for free. And so when someone say, hey, Kelsey, we want to advise, I say, stop this. When I first started, it was like, I need some equity. So advisory shares. But I understood something different there. I said, look, I might need a quarter point, half a point, or a whole point, depending on where you are with your funding rounds. How much risk am I taking giving you this time? And how much impact will I make? Let's say I asked for a quarter
- [128:24]
Interviewer
point of a company, quarter point being 0.25%.
- [128:27]
Kelsey Hightower
Yep. Of equity. And I would say, look, you know, I used to think you can be advising for four years. That makes no sense, actually, in my opinion from experience, advisory, I think, is really good for like a year at A time. Because advisory should have impact. It shouldn't just be like this, hey, let's talk about what we're doing. And you just give superficial advice like, no, it should really make an impact. So if you're going to make an impact, then you've earned the equity. And so what I started doing, instead of a four year vesting, I would say, look, I need one year, no cliff, ten year exercise window, never losing again.
- [129:01]
Interviewer
Not on early exercise, not on earth exercise.
- [129:04]
Kelsey Hightower
That worked really well. So you look at Carta and it's all stacking up. But I realized that I wasn't necessarily avoiding the working for free problem because if they don't have a good exit, you still get nothing. So I started adding the retainer component and the retainer component, I used to think about them as dividends, right? So you may give a dollar amount, it could be 1500, it could be 3000, it could be 5000. And you get that every month for that one year. And so in your mind it's like, all right, I get $60,000 plus equity. And now it's like, all right, you should probably call me because it's just a very expensive person sitting there. I remember the first time I had a decent exit. I advised the company PixieLabs. They were doing observability with a small twist on Kubernetes. They were doing as observability. And what they were doing is leveraging EBPF so that way you didn't really have to add any agents or instruments. So the way EBPF works is you almost get to the kernel level. So if two applications are talking to each other at the lower level, I can actually see that networking traffic. I can see what port is bind to and I can also see that it's a go program. I can even walk the tree like a deburger would. And so they were doing observability this way and they had, you know, almost like, hey, we're going to compete with datadog or something like that. And as an advisor I looked at it and said, look, you could try that. But most people are really not interested in changing observability stacks just for a slightly different way of doing things. Even if it is ebpf, they're like, what do you think it should be? So they're in stealth mode. Their investors are like, yo, it's time to come out of stealth. We got to start, you know, getting some revenue from this thing. I said, hey, we need a few more months and we need to take another Approach. I'm sitting with the team and I'm like, you know what you have? You have like an agentless thing. And they also have this thing called Pixie Scripts that allows you to kind of make your own dashboards or aggregate your own metrics. And I started like, oh, if you're a system administrator, you can wrap them like command line tools, and you can, like, create a thousand clusters with these Pixie scripts. And so this idea that we should pivot just a little bit, change the messaging just a little bit before we come out of stealth. And they gave a couple of presentations with this agentless observability. And then we did a keynote, or we did like a Pixie Day before coming out of stealth. And we showed system administrators this new vision. It went well. I did this opening keynote, I showed the vision, I interviewed the founders, and like, why did this need to exist? Why not just Datadog? The next day they had offers. I think one of them was for maybe VMware, but the other one was for New Relic. And they got acquired by New Relic. And I was like, wow, what does this mean? And I remember the lawyer came and said, hey, we need to accelerate all your shares and this is the money we owe. I was like, whoa, this can work. But also I felt like I actually made an impact. We did Pivot, and then the way it works in advisory with VCs, they like returns, and word gets around. Kelsey's advisory can have impact. So other founders are coming. VCs are recommending some companies that need help. So over time, you know, some founders reach out. Like Guillermo from Vercel reached out, like, hey, Kelsey, this is when they were making their pivot from just being purely serverless and front end to thinking about going a little deeper in the stack. So I spent a couple years with the Vercel team. Docker can go on and on, and that helps you build out a portfolio. And so I was like, you know what? Stocks are cool. Remember, I started to divest from that once I have my retirement plan in place. I was like, but I do like the concept of the entrepreneurial mindset. And so helping these companies get to the next level, even for my little short amount of time, the little small impact and then being able to share in the outcomes became a major part of my advisory work. So my advice to anyone that wants to do advisory work, it really helps to be a domain expert. Deep. So if a team is about to build out their engineering team and you've been an engineering manager or a team lead, don't just go say, oh, when we were at Google, we did it this way. That's not what a startup needs. What a startup needs is you to say, listen, as you build out your team, you have to think about growth trajectory. You're going to have to think about vesting schedules. You have to think about personality types, impactful work, junior versus senior spread, when to bring in engineering, leadership, when not to. That's the type of advisory that can help accelerate a startup from one stage to another stage. So then that might be your type of advisory role. So when you meet a founder, say, hey, listen, here's my domain expertise, here's an impact I can have. And if you think that's going to be good for you at this stage, and don't get offended if your advisory is no longer necessary because maybe they're going to move on to something much different than what you're good at, let them go get a new set of advisors because you've done your part.
- [133:46]
Interviewer
Yeah, I think this is the part of the lowy ego. Right.
- [133:49]
Kelsey Hightower
Exactly.
- [133:50]
Interviewer
One thing I really appreciated you especially, this was very visible after you retired that you took intention to understand technologies. You had some time with crypto where you went deep and tried to understand it and asked really good questions. The community response was a little bit weird, I think hostile, but I'm not a fan of that specific community. But the other thing that struck me was with Genai as well. You, of course is everywhere now. It's impacting. Everyone is using it, trying to, trying to figure out how to best use it. How have you gone about understanding Gen AI, especially with your approach of like, all right, you know, one step at a time.
- [134:28]
Kelsey Hightower
I'm very much a people person all the way through and through. I'm very in tune with myself again, when I'm cleaning, I'm reflecting. And so this whole game to me feels like it's not about the ones and zeros. I know everyone wants to make it that way. We judge too much of society based on this. If you're a billionaire, you're automatically getting respect. If you have no money, people walk past you without a second look. And it's unfortunate that it's that way, but I really do think about it. And the weird thing is when you say you think about people, people find that very odd. If you're mean, if you're narcissist on Twitter, that's normal, that's expected. If you want to get over people, if you want to game the system, that's normal. Almost expect It. But when you say, I want to just be kind to other people, that feels weird. People are like, what the hell is this? No, no, no. We're all just gaming this thing. My philosophy around technology really is this people first situation. So when crypto came out, everyone was like, oh, we got these tokens, blah, blah, blah. Kelsey is open source, you know, blockchain. I said, I don't doubt any of these things. I've been a part of open source movements. I've been a part of things that maybe have threatened other people's jobs. I get it. But this crypto thing, I can't help not think about the financial system in a way that impacts real people. Real people are forced to work, right? So we go back thousands of years. You can go out into the forest, get something to eat, and that's it. Now you have to get a job because the forest is off limits. So now we force people into the cycle. This is reality. Now you're coming and saying you want to change the currency, right? Not all of them, but some of them did. So I'm not as concerned about how blockchain crypto, that's not as interesting. You're saying you want to change the currency. There have been countries that have gone through currency resets. This is not a nice experience. Because if you had a little bit of money, a currency reset can mean you have now no money. If you had no money, it may feel like it's impossible to ever get any more money. If you're retired, what do you do during a currency reset? You don't have any way of making new money. You're retired now. So these are very, very real things that I thought that group of technologists were ignoring because money go up. And so when I would have those conversations with them, they wanted to debate the low levels of, like, crypto and how transactions are settled and, you know, things about encryption of blockchain. I was like, yo, that stuff is fine. And we can go back and forth, but at some point we have to talk about how it impacts actual people. When Genai comes out. I'm now super into the philosophy of everything. So there's one part of this that's a little personal. Of course. We spent all of our careers learning all these skills, training our own models, right? We've learned to program. Yeah, we learned to program. And it's hard. Well, not just hard, but there's. There's aesthetics to this. It's not just blindly typing code. There's an art form to it, right? This is why we have Ruby. And when you read about the. About Ruby and Mat's vision for having something that can be almost romantic to write. Perl has its own subculture, Golang has its own culture. So we're not just writing code. And my entire career, I always thought about writing code as decision making. So before we do anything, we all figure out what needs to happen, and then we have to convince the computer to do it. And every keyword, every if statement, every function call is a decision we're making. And of course, the syntax kind of gets in the way from time to time. So stack overflow, we go. So Genai comes out and early stages is kind of like people are just talking to the machine. And I've never been impressed by talking to a computer. I'd rather talk to real people. So I don't really care too much about that part. Yes, it mimics human capabilities. For people that want to talk to a computer, knock yourselves out. But then we get into the code generation piece. Now we're back to where I was with the crypto stuff. I've used the compiler, it generates a lot of code for me. I've been doing that for a very long time. I haven't written any machine code. And I post things like, hey, I'm adopting the zero token architecture. People are like, what's zero token architecture? I was like, instead of burning tokens, you learn things and you think for yourself and just complete tasks. And they're like, oh, why would you want to do that? It's like, because we taught the machines. I don't know why people skip this step. Hey, Kelsey, there's going to be this artificial intelligence going to do all the things like that. We trained it. So all those times I'm writing code, the books I've published, the comments back and forth on helping people solve problems, it's all in there. Maybe it's arguable that they have their own worldview based on that, and maybe it's slightly different, but I can never put the machine over a person under any circumstance. And I think there's a subset. I don't want to say everyone in this space is doing this, but there is a healthy subset of people who really believe what is the purpose of a person. Why do we need them to write code? Why do we need them to build software? It's like, maybe you don't understand what the job has always been. We are trying to solve human problems and we use whatever technology is required. In some cases, the technology happens to be software, and software ain't required for every human endeavor. And I think a lot of people are just in this bubble where they believe software is the only way to solve any problem. And then they think Gen AI solves all human problems. And this is where I start to push back on that narrative. It's like you learned to love this job and you forgot what the rest of the world is doing. And so I feel like some people are now trapped. And that person, again, I keep going back to that person walking into the industry for the first time. Luckily for me, they were looking for people that had skills and there was a pathway, so many pathways for us. Now the new generation that's coming out, they're unsure of themselves. Hey, I'm watching the news. You guys keep celebrating people just using Genai to do everything. What am I going to do? And I just can't accept the answer being you're just going to come in here and use Genai to do everything and all of you are now just the same.
- [140:39]
Interviewer
Okay, so this is the one that you cannot accept. But of course, through advising, through meeting people to talk, through talking to people, what are some of the promising parts that you might see or even parallels to previous technology revolutions? May that be kubernetes or may that be the as we were moving to a lot more powerful computers when you're coming out.
- [141:03]
Kelsey Hightower
So the way I think about it, so for example, if I'm doing due diligence or the fund that I do due diligence for, that means before you make that decision to write a check and explain to our LPs why we took this position, we need to do a little due diligence. And we. And the way I do due diligence, I want to meet the founder. I would like them to walk me through the particular product and I go one step deeper. Let's look at the code, let's look at your Amazon bill, let's look at the architecture, let's look at GitHub. How do you manage issues? How do you all work together? I want to get a sense for the team, the product and its trajectory. And when AI is involved. The one thing I just do before the thing kicks off in this meeting, do not say AI because what we don't want to do is use a big umbrella to describe what you're doing. Let's get concrete details. These are computers. These are computer programs. Yes. Just like when I saw a regular expression for the first time is a different way of thinking about software than, you know, imperative things. If else then so I Get that? But now you have to show me what you're actually doing. So when we do that, when I put that handicap in place, now they're forced to show me the problem they're solving. They don't just say, hey, AI for healthcare. Nope, show me exactly what you're doing. And so with that handicap in place, the really good founders, really good technologists, what they do is they say, hey, here's a problem, and here how an industry currently solves the problem. And here's the drawbacks from that. And since they can't say AI, they can't say agentic. They just have to show me how they make the problem better. Now, as a technologist, I know that if I gave you a random PDF, there is no easy way for you to procedurally program parsing PDFs. You just can never anticipate everything you will ever see. But it makes sense to use ocr. Or if you don't know what an OCR is, you might say, I will go use Claude to do it, whatever. There's probably an easier way to do this than using a large language model. There are smaller models, smaller techniques, so my advice to them would be, you know, you don't need to use an LLM for this. There are smaller models or smaller AI techniques that are not gen AI. So they're, oh, good feedback. We can probably lower cost here. But then when they take these technologies and they do something novel with it, I'm like, you know what, that is a good product for the people doing this work. And I'll say, you see that you did that without saying AI. So on your website, why are you burying all the value of this platform by putting AI in big layers before we get to the value? And so maybe they do change the website to actually talk about the value. When they do the demos, they start with the value versus hand wavy text boxes where they put in a prompt and it does something magic. Physicians don't want to do that. They want to see some of the other contextual things and leverage AI to bridge the gap between what they're currently doing and now what's possible. So my advice to people that I'm advising, you know, all the startups, I gotta make an AI pivot. I said, if you all pivot to AI, then you all will have a problem. It's like kids learning to play soccer. They all run to the ball, no strategy, spread out, figure where you add value and play your position. So when I'm advising a startup, what is your position in this big Landscape, you can't all run to the AI ball. You gotta stand back and figure out which value you're going to add. So one of the companies I Advise, they're called MassDriver. They have like a visual kind of infrastructure as code. You take Terraform, you give it some metadata and then you can interact with it visually. And part of that visual interaction allows you to do things like this app needs this database and just drawing that line allows the credentials to flow to the other app under the hood. And now you have a config.
- [144:38]
Interviewer
Awesome.
- [144:38]
Kelsey Hightower
So now Claude is the big thing. Everyone's like, no, I don't need that, I don't need Terraform. I'm just going to use cloud to manage the cloud. Now someone would experience, I'm like, this is about to be real fun because I've seen what humans do when you just give them AWS console. Watch what Claude's gonna do when you give it to AWS console. And so knowing this, it's like, okay, here's how we can add value to this. Number one. We can take the things that you have in this visual box and split it up into value props. Number one. In order to show things visually, you have to have context. So let's call it the context engine. And then that context engine can be queried by Claude. So instead of pointing it aws, it's now pointed at only the resources you use. And for people to understand why this is important, if you take some of these agents, they just start investigating the console, like, ooh, what's lambda? Nah, don't need that. But Lambda now is now running. What's this? Law Balancer. Oh, don't need that. But now that could be running and you don't even know the mess that it made.
- [145:45]
Interviewer
They don't even know the message made.
- [145:47]
Kelsey Hightower
It doesn't even know the message made. So now we say guardrails or give it context. So I said, oh, okay. So instead of doing a full on pivot in a naive way, let's reposition here. So now some of the features we have become guardrails. Some of the things, the way MassDriver does things with IC or infrastructure as code can become skills. So now if you bring Claude to the scenario, instead of starting from scratch, we can just allow the agent to interact with this platform the same way humans do. And then what we end up with is if a human wants to interact visually, it works. If you are just like, want to have some automation in your pipeline, then just call the APIs and you still can interact with the context and the deployment engines. But if you really just want to use CLAUDE code, then now CLAUDE can interact with the same guardrails and structure that the rest of your system does. And then CLAUDE becomes a little clearer. We're not asking it to be magic, we're asking it to be an alternative interface for getting something done. Skills MD becomes implementation detail. And then when we do a webinar where we present this, just watch the light bulbs go off for the people watching and for the team. And that to me is the type of advisory where you can look at Genai. I'm not just like a Genai hater. I just don't like the naive promotion and adoption of it. I think it should be way strategic. And since I think about Genai as a tool versus the great human replacement, then I can use it in way more pragmatic ways.
- [147:18]
Interviewer
Thinking about it as a tool, what capabilities do you think it gives us? Software engineers specifically? And also maybe what are some areas where it can maybe give some overconfidence.
- [147:29]
Kelsey Hightower
So one thing I've tried to do is to be a little bit more positive in my thinking because it will be very easy for me to go down the rabbit hole of identifying all of its flaws, right? Like, oh, look, it makes mistakes or it hallucinates sometimes, or it, overly confident, gives me a config that then blows up production. I'm going to assume those will get slightly better over time or human in loop will catch those things. So I'm going to give it, for the sake of this discussion, I'm going to give it a little grace. The things that I think it should do well, that I think don't get used often enough. There's this concept where people really think it makes sense to do inference every single time. For example, if a human writes a piece of code, we'll write it once, right? Let's say I wanted to authenticate to an endpoint. So I will call the code, I'll go look at the documentation, go to stack overflow, figure out the example, I'll call it once. If I had to do this twice, it's probably going to become a function that I call multiple times in the code base. If I do this like five to ten times, this may become a library that I import in multiple apps. So that tends to be the human flow because there's no need to infer or write from scratch this particular thing. And we've done that. That's where open source comes into play. And when we're really doing A good job. These things get just baked into the framework and they're just there, sometimes even in the os. In the os. And then sometimes things like encryption find its way all the way to the hardware where we do offloading. So this has been the loop that software engineers go through for a long time. Gen AI to me should be no different. So if you find yourself generating the same blocks of code over and over, even though it feels fast or convenient, it is still insane. Yeah. At some point you should say, whoa. Why is cloud generating the same block, copying things everywhere? Because we know where this leads.
- [149:18]
Interviewer
Yeah, we've seen that. Of course we do.
- [149:20]
Kelsey Hightower
Yeah. And then people get excited. Well, COD can refactor all of it's like. But that's a waste of energy. There's no reason to do it just because it can. So I hope what developers are really realizing, what I realized myself from looking at this is the number one thing I realized, is that most of our APIs were designed incorrectly, even for humans. Right now, a lot of our APIs. If I were to think about infrastructure, you have to call like 7 APIs to get a VM in the cloud, create a VM, a network, a storage device connected to a VPC and then attach credentials to like. That's not intent based. I want a vm. So the first thing you kind of see from this movement is things like MCP where we wrap these imperative calls into an intent based thing that says create a VM and then that reflects out to the other ones. But here's the thing. I saw this before that. I saw this with Kubernetes. Kubernetes does the exact same thing when you say give me a ingress or a service in Kubernetes that reflects out to seven calls too. To create load balancers, SSL certificates, DNS setup.
- [150:19]
Interviewer
It just hides you behind the interface, Right?
- [150:21]
Kelsey Hightower
Exactly. I've seen this before. So I'm looking at this like, guys, the fundamentals of this isn't like everything is going to change because of mcp. What the hell, A whole conference, Guys, this is just API design. So in this particular case, I hope developers realize that we shouldn't have fought so hard. RPC versus rest. There was this big tug of war around composable APIs, like 2000s. Yeah. And like a lot of people went down this rabbit hole of like Create vm. Oh, that's a. It's not flexible enough. It's like, so what? Create VM V1, create VM V2, who cares? Because these are intent based APIs. So I think these new tools are reminding us of this. When I look at the way people are prompting and how they write their prompts. A lot of our programming languages were too rigid too, to express what you really wanted to happen. We were so afraid. Maybe Ruby did a better job than some other languages where they try to give you things like until. So that way it just flowed better as you were writing the code. Other languages are a bit more rigid. It's like, what does FN mean? So you have to go look it up every single time. So writing code become very laborious. So when people write a natural language prompt, it kind of tells us a lot about the intentions of querying a database. So I'm hoping developers learn better API design from that front. The other part was I still get frustrated. I want to learn new technology. I go to the website, there's a little bit of documentation. But for some reason, developers still write documentations as hints, as clues. Right? I'm trying to learn a new programming language. I won't throw any of them under the bus. And I go there and I'm just going to go look at the standard library. First thing I like to do sometimes is just like, I'm going to parse a JSON file, get a feel for the language. And there's like, there's a JSON library. Sweet. We're off to a good start. Click, and you look at the documentation. You just see function calls. I'm like, can I just see a working example? What do I import? What do I put in this thing? What comes out? And then maybe what do the errors look like? I just want to see a full example. They're like, no, you just get reference. So then what do we do next? Then we search the Internet and you might land on Stack Overflow. Someone's blog, did they give you a full example? So now I got to copy and paste it and see what this thing does. And hopefully it's up to date with the actual documentation. This has been the loop we've been going through for so many years. So when I use things like Cloud or various tools, they close that loop. You get the example right here. And let's not pretend that Stack Overflow examples were perfect. They were not. So to me, like, having something where the tool would then try to build it to tell me if this is correct syntax, or thinking to make sure that it's giving me a good suggestion, even I can say that is an improvement. But I hope we learned that maybe we should not just give hints in the official documentation, I'm watching people write these huge markdown files to give the agent context. How about you write documentation to give me context so I can have full working examples?
- [153:22]
Interviewer
Now with agents, we could actually generate documentation a lot more intentionally. So like as devs, most of us just don't like. I'm not sure if there's anyone who likes writing documentation. We don't like doing this, we don't like writing tests in general. These tools can help with that.
- [153:38]
Kelsey Hightower
But this is where I think we tend to make a mistake. I remember when I was learning Java for the first time and the Java developer was like, we don't write docs. The code is self documenting. I was like, no, it's not. It's just documenting hints. I still have no context because to me, again, software development is to me a human endeavor assisted by tools. And so there's a style to documentation, there's a personality to documentation. It's like a movie, like you're trying to educate a person. I don't want just hints saying, hey, this thing exists for these reasons. We conform to this particular specification. There are multiple ways to write this code. Here's the most popular way. Here's what bad code looks like, here's what high performance code looks like, here's when to use this library, here's when not to use this library. I want that kind of in depth as I'm training my own model. And what I'm seeing now is, which I think is a good thing, people are writing a lot more documentation to be consumed by the agent, to give it context. I was like, man, I wish we had the same motivation just a decade ago because I think a lot of us would have been way more productive if we didn't have to try to do a wild goose chase every time.
- [154:47]
Interviewer
Well, interesting enough, we. Kat Cosgrove told me that one of the reasons she thinks Kubernetes one was documentation, they take it extremely seriously. Few other project, if any, does it at that level. So just proving a point. One thing that a lot of experienced software engineers are worried about right now is all the AI. We're all using AI agents. They do generate code really quickly, which is something writing code used to take a lot of effort. It took a lot long time to be good at it. Now with code reviews, there's now more tools coming in and some people are like worried like, okay, like, what is happening to my profession? Like, the craft of writing code seems to be something that we can offload and more and more people are Offloading, including prominent people. What will this do to software engineering? And what advice would you give to people who are experienced software engineers? They're a little bit worried because it's a big shift. They still want to, you know, like, be whatever great engineer will look like in the future, but what steps might be able to take. And I especially like your take because I think you're pretty grounded in just looking at this from a vantage point that some of us are not.
- [155:54]
Kelsey Hightower
I think as a software developer, the first step you have to do is have a bit of reflection. For the last 20, 30 years, you have been automating a lot of industries away yourself. All those programs I remember seeing, maybe it was like a diagram of every device that has been replaced by the iPhone, the radio, the calculator, the compass, all of these tools people used to buy individually. The top 30 of those electronics from the last 40 years are all in your iPhone, all of them. That means some electronic makers have gone out of business. They're gone. You did that, not in a malicious way, but you were part of that. And so the software developer has been glorified for a very long time. The Internet, some people would say, caused the downfall of magazines and newspapers because of the convenience of having a software approach. And so you have been part of the change to other industries and other people yourself. What did you think about that? Did you even think about it at all? So let's not be surprised that you find no sympathy from all the other professions that you've helped force change upon. So I think that's step one. You have to go do that reflection, because if you don't do that reflection, you won't know how to behave. Now you're going to be complaining and people are going to look at you crazy, because where was this empathy before? You might be very excited about this and not realize you're only excited because you're in position to benefit from this. So if you work at an anthropic, of course this is the future, because it's in your hands. If you're at Nvidia, of course this is the future because you will be selling the picks and shovels. And so you got to ask yourself, why am I excited now? What I don't want you to do is necessarily feel guilty about it. But I need you to see the big, big picture. It's going to help frame everything else. The second thing I want you to do is ask yourself, what was my job? Remember, there was a point in my career where I was lying to myself. I thought my job was to be the best Linux administrator ever. You as a software developer, you may have thought your job was to be the only person in the organization that can write code. And since no one else could do it, you are safe. Right. And so you didn't learn any other skills? Networking, product management, design, talking to customers? Nope. All you had to do was write code and you were set and you probably made more money than everybody and you were fine with that. Now you got caught. The only thing you were good at is now been commoditized and again you did this to others. So let's say the vision you have for yourself is only in this very narrow realm. You're going to be very afraid of this trajectory because all you know is software developers write code. That's it. Some software developers still don't write tests, still don't know how to deploy anything. And so they are really afraid because they can't see any other way that this plays out. Now if you're a Full Stack engineer, you're probably like, man, there's so much more than just writing code. You have to do architecture, you have to do design, you have to do so many other things that I love Claude, because now I can focus on those things and I can use these tools instead. So I can see why that person would have that perspective. Now I understand why that Full Stack person has a perspective of watching the same people commentate that the code generation piece replaces everything else. They're going to be like, no, you don't know what this job is. It's way more than just writing code. Writing code is the last step. If you're a security engineer, you're probably like, we never figured out security for the pace of the current enterprise. The one before, yeah, like everyone thinks they're moving slow. I remember I took a security training thing and most of them aren't that good because they can't go super deep. They just tell you, hey, here's how to avoid phishing, here's how to not leak information, adhere to various laws and things like that. And then they said one thing this time that I learned that was pretty good. What's the key to protecting yourself? And they say, you know what, just go slow. A lot of attacks are, I'm about to board a flight and let's say you're an admin or you're a VP and the CEO texts you right now we need to wire the money to Oracle to pay for the license. They're going to cut it off right now. This Needs to be done immediately. You look at your phone, it's definitely from the phone number of the CEO. You have a good relationship. Everything looks right and it's moving fast. So you're like, man, I'm on a 10 hour flight, I need to do this now. Turns out the attacker knows you're about to abort this flight. They've seen all your previous text messages. They know how your manager talks to you. They know that you've moved fast in the past. And so they now are primed to get you to do the exact same behavior again. And you could be the VP of security. So you should know better. And sometimes that naive confidence will make you feel like, I'm obviously not being phished. I've done this many times. This is definitely the CEO who would know how we actually operate and who would know that I can actually do that. So what do you do? You'd make the transfer. And just like that, 10 million has been wired to the wrong place because you moved fast. It wasn't because you were not smart. It wasn't because you were not productive. Because in this case, you were productive. But you did the wrong thing. So when I think about code, there is value in having a healthy pace. Let's say you're an insurance company. You sell insurance. Hey, make model, how old are you? Have you had any accidents? Okay, here is your insurance for the year. Simple, very simple thing. If you're an insurance company and that's all you are, you're kind of close to being done. Now you could say with Genai, we should get into payments. We should compete with Doordash, right? We have all these tools. Let's go.
- [161:55]
Interviewer
We could build it.
- [161:56]
Kelsey Hightower
Yeah, we can build it. But the thing is, should you just build it? Because you can. And the answer is typically no. So we usually optimize ourselves as humans around the pace needed for the task. And when we don't need to do that work anymore, we move on to something else. So now I think what we're going to end up with is people not realizing a lot of this stuff we were doing in software engineering was decision making. What database to use, what schema. Should we really collect someone's Social Security number or should we avoid it? Not. Yeah, I can write code to parse a Social Security number really fast. Like, no, no, no. Should you even do it? And so when you write code, it almost makes you slow down again. Because there's been times where I thought I had a good design. There's that phrase, writing is thinking. So is writing code. So as you're writing the code, you're like, hey, wait a minute, this loop is ridiculous. Right. Not only is it going to make the computer warm, this is not the right thing to do. There's a better data structure than the algorithm that I'm using. So then you stop and say, hey, the data structure is wrong. We need to change the way we print receipts on the cash register. Sure, I can write this code, but this is the wrong data structure. While I can generate the code, doing reports are going to be a nightmare. Summarizing this data downstream is going to be a nightmare. Stop everything. Now that I've thought about it, we need to change the architecture from the top down. So decision making sometimes does benefit from slow. And when I'm saying slow here, we're not talking waterfall. Six months.
- [163:27]
Host
Yeah.
- [163:27]
Kelsey Hightower
No, we're just talking about maybe one more day before you go at it. And I think some of us are going to miss that part because cloud. Spit it out. Ship it.
- [163:37]
Interviewer
Yeah. And that's also one thing that you always have the more experienced generation be worried about, the young generation. I remember when I joined the industry, resharper had come out resharper and the experience of the old guard was like, nah, you're not a real developer if you use resharper because you're not going to learn the library and you need that and that's what makes you a real developer. And then I remember when I was now five plus years of experience and Stack Overflow started to become big and I was like, nah, you don't want to go to Stack Overflow because you're not going to learn the real thing. But now what the current old guard is saying, which is, I guess I'm part of it, is like, well, if you use AI, you're going to miss learning the basics. And when you have learned the basics, it's so much easier to use AI. And I wonder if we're just repeating the same mistake as the previous ones did, which is the new generation usually figures out the tools they understand how to do it. Or are we rightly concerned that some people who are coming in as AI native, they're now learning to code, they can jump through so many layers that they will just not see what's under understand. Or are we just making assumptions that might not be true?
- [164:50]
Kelsey Hightower
Here's where I think it can be right on both sides. Do you need to learn how to code to make an impact in this industry? The answer is no, you do not have to. There are some People who use these no code platforms where they drag and drop and they produce a really good app. There are some people who have built a consultancy business by just using Wix, right? They go there, their website actually looks pretty good. And so they got really far with that. Now, for what they're trying to do in accomplishments life, they'll probably be fine. But let's just say you are a software engineer. And the idea behind Software engineer is not limited to just producing apps. Software is the interface between hardware and things people want to do. So there's a whole bunch of things you need to learn. So if you want to be that type of software engineer, you're going to learn hardware too. If you don't understand hardware, you can never work at that level. And look, if that's not your job, then so be it, but you'll never have that creativity. I remember seeing someone was like, hey, you can do isolation without a vm. I was like, how would you do that? He was like, oh, because when you boot the kernel, there's a thing you can do before the kernel loads to isolate it in a way that you can lock down processes. The only reason why this person knows is they know the full boot sequence from firmware to switching to the kernel, and the tricks you can do in between. Now, for me, that doesn't work at that depth. I'm thinking there's only virtualization, CPU isolation, things like gvisor where you intercept system calls, but never did I think about the boot sequence. And so yes, you can get very far. But as someone like, we applaud every version of OPUS that's released or chatgpt, but there are versions of yourself that get deeper from these new trainings. So no, you don't have to. But if you ever want to get better at anything, and sometimes that depth, that nuance, is the thing that leads to invention, right? If you know how a compiler works, if you know how memory management works, that might give you enough information to say, oh, I can make a new programming language if only thing you know is the Surface. You can't even imagine how you can create another programming language that is better fit for the task at hand. Because you've never gone that deep. I'm not saying everyone wants to do that. So I think it is fair to say all I want to do is come in, get a job. And if that job can be done by using AI tools, I think the side effect of that is then that job will be commoditized. It has to. That's just the way it's going to go. But I've always seen myself for my entire career, I want to learn more. I want to go deeper. I want to go so deep that I can create. And I think a lot of people who are doing this, the reason why we're having this reaction, some of us, some of us, part of our careers have been the creation part. There is no spec for this. There's no protocol for this. We're going to make it work. A lot of people that are doing like the reverse engineering and the hacking, they're like, there's no framework for what I'm about to do. I just know how memory works. And I don't care what your security tool does. I will make it do what I want it to do. You need to go way below the surface. And so for a lot of us that are saying this, we know the value of the fundamentals that lead to the other stuff. And so if you tell the next generation, oh, you don't need to learn these things, it's like, that may be right in the short term, but we know for a fact your career will be limited. And that may not be a problem. You have to decide. But make no mistake, if we put this much effort in training the model so that it can spit things out, you better make sure that you are willing to train your own model. So my advice to people would be, and maybe we should talk about it different. Maybe we shouldn't have so much fear mongering around it. Maybe we wouldn't should put it a versus this. We should just say, great artists tend to know how to mix colors and it is in your benefit to understand the primary colors so you can mix them to get the other colors. And it's a superpower, right? So you don't have to go buy. You know, imagine an artist trying to go buy 16 million colors and put them on the desk because they don't know how to mix colors. If you teach a person how to mix colors, you can get any color you want. And I think that's the way we had to approach it. It's just another skill that if you had it, you might just unlock some creativity. So I encourage you to learn it.
- [169:03]
Interviewer
Kelsey, thank you very much. This was just an amazing conversation.
- [169:09]
Kelsey Hightower
Awesome. Thanks for having me.
- [169:11]
Host
I will admit I was glued to
- [169:12]
Interviewer
my chair for the whole of the conversation.
- [169:14]
Host
Apologies that it took this long, but I hope you agree that this specific
- [169:17]
Interviewer
one was worth it.
- [169:18]
Host
Kelsey's past was just so unlikely. He's someone who was raised by a single mother, dropped out of college in favor of installing DSL lines door to door at 19 and still ended up as Distinguished Engineer at Google Cloud. Only a few hundred people who hold that title at the company and he retired at the top three years ago when he decided that he no longer needs to work for others. What an inspiration. One thing I took notes on was when Kelce said how every job is an interview. When Kelce was giving the Gaufercon talk and PX Taxi booting Coreos from his slide deck, he had no idea that the Coros team was sitting in the audience. And that's how he ended up at Coreos when he was contributing to Puppet at nights and weekends. He didn't know that James Turnbull would walk into his office one day and recognize his name. He just kept showing up and doing
- [170:00]
Interviewer
the work in public.
- [170:01]
Host
It's a lesson worth remembering. Do the best work you can at work. It might unknowingly be your job interview for your next step in your career. I also found the Microsoft Microsoft offer fascinating.
- [170:11]
Interviewer
Kelsey did not use the offer from
- [170:13]
Host
Microsoft as an ultimatum at Google, even though he could have. He just told his manager the truth
- [170:18]
Interviewer
and then Google mashed the offer.
- [170:20]
Host
Obviously at this high level there's no universal composition negotiating advice that always works. Being a straight shooter with high integrity is something that is good to keep in mind. I was also inspired by Kelce's focus on minimalism. He treats money as freedom tokens and made sure that his lifestyle style never inflated with his salary so that early retirement was always a real option, not just fantasy. Finally, Kelsey's AI takes his point is grounded and pragmatic. AI does not change what software engineering is actually for. The job was never just to write code. The job was and is to solve human problems. The engineers who understand this are going to be fine. Do check out the show notes below for related the Pragmatic Engineer Deep Dives on Kubernetes and other related topics. If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also
- [171:06]
Interviewer
leave a rating on the show.
- [171:07]
Host
Thanks and see you in the next one.