B (2:54)

So we have customers writing in with things that are broken every day. I actually broke a piece of the creation flow yesterday in the live website for like 25 minutes. And so I got 10 support emails during that time and I had to then respond to all the support emails after I fixed it. There's half of my time right now is sort of like dealing with stuff like that. And then we try to spend the other half thinking about like, okay, here's the next big thing we want to launch. That sort of changes across the whole flow of someone signing up into the product. We want to sort of like make something major.

A (3:23)

Consumer tech is constantly changing and so is the way it's rolled out to consumers.

B (3:29)

The ancient philosophy of the 90s and shrink wrap software is like, referred to as like a waterfall style of development. So you sort of like make all these changes on your local computer. No one ever sees them, you know, until finally the team is ready and you're like, you've put the last little like touch on it and then it goes to like somebody else who runs qa.

A (3:48)

QA is quality assurance. They're the people who test the software before it's finalized and then you release.

B (3:54)

It to everyone in a box and.

A (3:56)

Then you have another version in four years after that.

B (3:59)

Yeah, exactly. And most people work now, you know, by basically trying to build all of the little pieces without breaking the current thing and kind of continuously pushing that out.

A (4:09)

That's such a good and helpful description. You want to keep adding the new things without breaking the old things. And there's just a natural tension there because every time you poke at the system, you're just taking the risk.

B (4:21)

Yeah, yeah. And that's the job. The people who are really the amazing software engineers in at least the kind of product focused world that I live in, are the people who can make massive changes that solve real business problems while also not breaking everything. The whole job really is sort of like you hold the structure of what the existing thing does in your head. It's like this big set of flowing gears and arrows and stuff. And then you want to just tweak like this one, and then you ship it and make sure that everything's still working and that your new thing is also working. And then tomorrow you do the next thing until all of a sudden you have a different shape at the end.

A (4:57)

So when you roll out new features or new Releases. Is it easy to get feedback? I can imagine people complaining. And I could also imagine we're so used to bugs that we just work around them and don't say anything about it. How do you get feedback that you can trust?

B (5:13)

Yeah, it totally depends on the scale and the type of company that you have. So one of my friends from a previous job just built an app that was an AI therapist. So they didn't know what the users were saying to their app because that would be an invasion of privacy. So they relied 100% on sort of like, hey, please tell us if something's wrong, because otherwise we're not going to know. In massively scaled consumer things like the Facebook app or YouTube.com, there's all sorts of mechanisms for measuring what people are doing. So you mostly rely on sort of the metrics in the experiment group that you're changing something in to reflect what you've done. So, for example, if you have the Facebook news feed and you make the like button, you know, the actual, like, thumb. If you make the thumb, like 50 times bigger than it is right now, so it's just this huge thumb, more people might click the thumb. So you'll probably see an increase in that group of likes. However, it might also be like, a terrible experience because you've got this enormous thumb in the middle of newsfeed. So you probably won't have as much continuous scrolling and usage because people will be probably mad that their newsfeed has these huge thumbs in it. So all that data will be reflected in the experimentation framework in the company automatically.

A (6:21)

That's really powerful. I mean, to have that level of detail automated, where you can kind of follow not just the immediate implication of, like, did more people click like or not, but also, you know, the next layer of did they spend more time on the site or did they abandon because it was annoying or whatever.

B (6:37)

Yeah, and then you have the like at Facebook, it was super interesting because you also had the network effects. So when my experiment group causes less likes on a certain type of content, that also causes that content to not travel as far or get as many views. Right? So, like, you can have really interesting intergroup experimentation results that are super hard to actually digest. So a lot of times what Facebook will do is turn on a new feature, especially if it's sort of making really interesting and hard to measure interaction effects, they'll turn it on, like, just in New Zealand. So New Zealand or Argentina or there's a bunch of countries that are commonly getting, like, the weirdest Facebook features first.

A (7:14)

So I feel like we have this relatively elaborate stereotype of a software engineer. Let me air it out and you tell me which parts are true and which parts are false. So it's someone very young, almost always male in the stereotype. Hoodied, wired on like, six monster energy drinks, weirdly in the dark. Why do they work in the dark? Often alone, they're kind of irritable, difficult to work with, but they're freak geniuses. They type at like 200 words a minute and can solve. So what's true, what's false?

B (7:51)

So generally the male, female split thing is real. And there's a pervasive problem with trying to get more women into the field that's trying to be addressed in a million different ways. But, yeah, current state of the world is plus 90% of software engineers are probably guys. So that's pretty accurate. I think the irritability and the sort of lack of social graces, I think aligns pretty well with kind of what makes you good at getting that whole context of the way that the system works into your head at once. You have to be a little bit of a weirdo to be able to kind of absorb that up into your brain all at once. And so I think that that probably aligns a little bit with people who are a little bit pedantic and that care about the difference between this way of doing something and this other way of doing something. And I think, like, that sort of co aligns with potentially being a little irritable. The other part that I think feeds into the irritability is there's kind of a weird thing which is that all of the business rules of the way that a company's software works, anybody who works at the company knows a part of them. But the software engineer who's adding the new feature has to know about every single one pretty much when they're in the code. So when you say, like, oh, let's just change that button from red to blue, you know, why not? That'd be so much more. Blue's more aligned with our brand. Like, that's a much better color. I think it's a warmer color for.

A (9:10)

It's so simple. I mean, it's just a button. Who cares?

B (9:12)

It's just a color, you know, and then the software engineer goes in there and says, like, well, you didn't think about dark mode, and you didn't think about, you know, in Arabic, it's rtl. So it's actually. It's on the wrong side. It's not Going to be next to the other thing that's blue.

A (9:24)

That's a great example.

B (9:26)

You know, then you dive in and say, like, actually, you know, a paid user, it's got this badge on it. And you didn't come up with the case of, like, what happens when it's got the badge on it? And so, like, you're taking everything that the business can do, and then you're. You're encoding it in all these files, in these workflows, and then you're like, you know, just, why is it so hard to change that? It's like, well, you didn't, you know, there's so much there.

A (9:46)

And it seems like it requires so much concentration, which I assume is why working in the dark, often alone, fueled by monster energy. Like, in my brief and unsuccessful time learning to code in college, like, I just remember these maddening wormholes where something wasn't working and you're trying to debug the code, and then like, three hours later you figure out, like, you're missing one close parenthesis or something.

B (10:11)

Totally.

A (10:12)

And that was both. The beauty and the terror of the job is that you knew it was solvable, but that didn't mean it was going to be an easy process.

B (10:20)

Yeah, so I do think that there's certain characteristics of. Is it like an ornery person who kind of enjoys, like, I got to figure out all these weird cases and then convey them to somebody and then kind of be a little bit pissed when they don't understand the complexity? So I think that resonates. I think the thing that's the biggest difference, and this is also the Hollywood archetype of like a hacker, like, hacking into the mainframe, and they're like, they're doing it on a laptop in front of you and there's like 10 people around them. And like, then there's like, the Russian guy's also hacking from the other side and like, it's a race. Or like the solo genius off in the corner, like, creating the best algorithm that the world's ever seen. That's crazy. Like, maybe that happens. Maybe, maybe there's somebody who, who does that. But most of our work is, like, it requires so much concentration that you really can't be under the gun. You really have to get time and space mentally to, like, figure out the plan, but it kind of can't be on the spot.

A (11:18)

Hey, folks, Dan here. Hey. Just for fun, I want to shout out to a couple of organizations that do work similar to ours in the sense of conversations with real non celebrity people that let you walk in their shoes. The first is StoryCorps, which you may be familiar with. If you want a glimpse into the lives of normal people who you'd never otherwise know, check it out. It is a treasure 400,000 plus conversations about everything under the sun, all searchable online. The link to their archive is in the show notes and also a shout out to the Story Preservation initiative that we came to know recently. It's about real people telling the story of how they came to do what they do, why it matters, and what they've learned along the way. For instance, there's a conversation with a children's book author that Matt and I really enjoyed. See more at the link in the show notes. And now back to our own show. I asked Taylor how you can prove that you're good at software engineering and how you can spot those talents when you're hiring.

B (12:36)

Well, it depends a lot on the in the environment you're in and like the actual interview process to try to understand if someone is good and you should hire them is very hard. And I don't trust my own interview skills at all because it's just so hard to actually tell why. You have a whole set of interpersonal skills which are important. So like, you know, can you talk on a podcast about software engineering or something? And that has almost nothing to do, but it has a little bit of something to do with being able to be functional on a software team. So like, if you are good interpersonally, you can be easy to discuss bugs with and sort of bounce ideas off of. And that can be very helpful for a team. But also it's not required for sort of like shipping a bug fix to production very quickly and being very detail oriented in doing that. So it's easy to sort of over pivot on the wrong attributes.

A (13:22)

Shipping to production, by the way, just means taking the code changes you've been working on and pushing them into the real world, like onto the actual apps that people are using on their phones and computers. And that can be dangerous. Taylor says even code changes from good software engineers sometimes cause crashes.

B (13:41)

One way to tell that I think someone's a great software engineer is like, first of all, just like, what have you done? Okay, you've been working for 10 years, tell me the 10amazing things that you've built, which isn't sort of like the cleverest solution to this problem or whatever. It's just like, you know, you're constantly putting things out there and the other side of that is like, okay, how many times have you taken down production and how did you respond to that? Because if you're not taking down production, you don't want to do it all the time. You don't want to be the bull in the china shop. But at the same time you're always breaking stuff just because it's impossible to sort of know ahead of time all of the possibilities of what the code might do.

A (14:17)

So it's almost like bad news if someone hasn't broken things because it means they weren't working on anything of substance. Is that what you're saying?

B (14:24)

Yeah, I mean, I think you're not pushing enough code to production if you're not also occasionally breaking production. The ideal state also, as you sort of like get more used to deploying changes, is how can you minimize when your code does go bad? So the real sign for me of working with someone who is awesome is like they figured out that the screen was melting before anybody told them and they already turned it off and they already have a fix in review. You know, the proactive, like self aware monitoring of sort of like the state of the world and then driving to fix it.

A (14:55)

So one thing I wanted to ask you about is in organizations there are natural tensions between certain roles. Like classic one is marketing and sales. And the marketing people are like the salespeople just aren't selling the product, it's brilliant. And the salespeople are like the marketing people built the wrong product or it's the wrong framing and you need to work on price and you need to blah, blah, blah. Where are the natural tensions with software engineers? If you're fighting with someone who is it likely to be?

B (15:22)

So in a software development team, typically you have a product manager at the center. They're the person who is kind of talking to the CEO and anybody in between them coming up with what should the product do next? Like what's the big feature? So for example, at Facebook, when I'm doing that stupid like giant like button experiment, it's like, oh, we want to make UI buttons like 50% bigger across the website because that's going to drive engagement, whatever. So the PM's coming with sort of like this strategy. The PM then usually works with the designer. And so there's a design team, maybe they have their own structure of how they do design reviews. But they say, okay, cool, the like button looks like this right now, guess what? Five times bigger. So they come back with like the pixels drawn out to actually show you what that's going to look like. They do a review with the CEO. The CEOs like, oh my God, I can't wait for this huge like button. And then it goes to an engineer. The engineer may or may not be in the room for that conversation also. But in old school Org, maybe they're not involved at all. They just receive sort of like the file that says here's how big the button's going to be. So the tension is usually between engineering and product because the product people want the button to be 50 times bigger. And then the engineer says, look, we can't make it 50 times bigger. We can only make it two times bigger because there's this comment button and then this, you know, in Arabic actually, then this way it flows, it doesn't go this way. So there's always a sort of like back and forth about the scope that you're going to take on between product and engineering.

A (16:44)

But the challenges of software engineering go well beyond organizational tensions. Sometimes the enemy is the code itself. And the consequences of a screw up can be huge. Earlier in his career, Taylor worked at a startup that developed tools that could be used by other app developers. So let's say you were building your own app and you wanted to include one of those pop up windows, like are you enjoying our app? Please rate it, blah blah blah. And then rather than build that feature yourself, you could just rely on Taylor's tools and snap it right in. So anyway, during this era, Taylor went home for Christmas one year and all seemed quiet, at least at first.

B (17:25)

Part of my life as a person who builds things on the Internet is that I'm constantly being notified when my things are broken. So I actually chaperoned a field trip the other day for my son and as I stepped into the school I got a page that said my site was down. So this happens all the time. I'm constantly waiting for the notification sound that I've been paged. So this service that was powering these pieces of UI in people's apps, I got paged on Christmas Eve. So I'm sitting around at my parents house, it's a party with 50 people. It's all people I haven't seen in a million years because I lived in San Francisco and I'm flying back to Chicago and catching up with everybody and my phone keeps dinging. So I open it up and I look at the graphs and basically you can see really plainly that we're basically getting five times as much traffic as we've ever gotten before right now on Christmas Eve. But the site was kind of healthy, it was pushing through. So I was like, okay, I'll have another Christmas cocktail and check in later. And it wasn't until the morning that I got the real scary ping, which is like, the site is completely unresponsive. We've got 10 support emails already about the apps are down. Nothing is working, you know. Oh.

A (18:35)

So Taylor starts trying to figure out what was spiking the traffic.

B (18:39)

Computer programs always have logs. So I open up the production logs to see what service is talking to me. And it's, like, not something that I recognize. It's something, like, roughly having to do with a race car or something. I'm not totally sure. And I keep looking, and all I want to do is get the service back up because we have tens of customers who are already pissed at me on Christmas morning. And I also want to go, like, open presents with my nephew, and I've got a headache. It's like a situation. So I figure out who the app is that's basically destroying the servers, and I just decide to turn it off. So I shipped a little change to our production environment that just said, okay, if it's this app, respond with kind of just an empty thing. And the goal was just like, let's have the site back up. Let's not deal with this until New Year's or whatever. So I do that. I go back to the Christmas festivities, and I just start getting a couple other pings. So I still look at my phone. You know, there's an email, you know, kind of ignore it. And it isn't until I get, like, the third one that's from the same guy. But now this time, it's on LinkedIn. And he's like, hey, my app is crashing right now. Every time you open my app, it's crashing, and I think it's because of your software. So I rush back to the computer, and I'm like, oh, my God. And it turns out that he had built this companion app for a toy, a race car app toy. And so kids were getting this race car from Santa and from Santa or whatever. It starts on Christmas Eve because some people get some early presents or whatever. Then on Christmas morning is this deluge because everybody's opening their things, and the first thing you have to do is you have to install an iPhone app to control it.

A (20:10)

Oh, no.

B (20:13)

So you open the thing, it's like, oh, my God, this is so sweet. I can't wait to play with it. It'll be so fun. It's gonna be so fun when you. When you connect the Bluetooth with the app. So that you can actually use the toy. That's going to be sweet. So then you download the stupid app, obviously, and then you have to like link up your sign in and whatever and then it just crashes. And so the children, you know, around the world are just like having this awful time with this. And it's. This goes on for like I was barely paying attention because I was trying to, you know, play with my nephew and stuff. While the bad fix was out there.

A (20:42)

It turns out, as is often the case with software, there was a compatibility issue. The app developer had installed an old version of Taylor software and when Taylor rolled out his quick fix, the old software didn't know what to do with it and completely crashed.

B (20:58)

So this is actually like the most embarrassing part of probably like anything that I've shipped, I think because it's just like so viscerally terrible. And it was like such a throwaway fix that I did to also cause the really bad part because it wasn't actually crashing until I did that bad thing when I basically tried to get our servers to shed some load and just shipped that sort of like crappy fix that was what actually caused the crash. So the whole app was crashing for the entire world for like two hours. And if you look up the Amazon reviews for the race car product, I did this a couple years ago, you can actually still find the vitriolic. You know, you ruined Christmas. This is the worst product I've ever bought. And it's. Yeah, that's like obviously the software engineer who installed our SDK at whatever the company is. Hopefully they aren't listening so they don't know it was me.

A (21:46)

So, Taylor, we always end our episodes with a quick lightning round of questions. Here we go. What is a word or phrase that only someone from your profession would be likely to know and what does it mean?

B (21:58)

Okay, I think a good one is tech debt.

A (22:01)

Tech debt?

B (22:01)

Have you heard of tech debt?

A (22:03)

I've heard the term, but I'm not sure I could define it.

B (22:05)

Okay, so it means that you want to get this feature out and if I want to support all of the languages in the world, like it's going to take four months. But if I want to get the feature shipped tomorrow, I have to cut corners and kind of like do the changes in a hacky way and maybe not test all these cases and you know, maybe the app crashes in. In Hindi or whatever. So tech debt is kind of like when you keep working like that, you build up, you know, a bunch of broken stuff along the edges and you need to, you know, eventually pay down the tech debt by just focusing on quality. So like, let's actually like, okay, we're not going to do any new, like, buttons, let's go back, make sure that the app works great and handy and sort of like resolve the tech debt.

A (22:45)

And so does the tech debt eventually get paid off or do you always just have a mounting pile of tech debt that never goes away?

B (22:53)

Depends. If you ask the pm, they're never going to want to pay down the tech debt. Just kidding. But yeah, like the antisocial engineer is going to want to pay that down and fix it up because they want it to be nice and tidy and the product people are going to want to ship the next, you know, thing that increases revenue or whatever. And so it's always a tension of, can we go back? Can we have a quality day? Can we do a quality week? In the big companies, you try to sort of build in some incentives for that. But yeah, it's a whole thing.

A (23:19)

What's the most insulting thing you could say about a software engineer's work?

B (23:24)

You write spaghetti code.

A (23:26)

Oh, what does that mean?

B (23:28)

That means your code is not well organized. I cannot understand it. It doesn't connect in logical ways. If you looked at like a nice, beautiful code base, it would be arranged in sort of like a nice flowchart. So I'm over here in Newsfeed, and Newsfeed needs to talk to the recommendations engine and the ads engine. You have these nice clean arrows. When it's spaghetti code, it's just a mess of, that's all talking to each other and calling each, you know, from crazy places. And you can't really reason about where to make the change because it's just all over the place.

A (24:00)

What's a tool specific to your profession that you really like?

B (24:04)

Using the one right now, and it's like a little basic actually, for being in the AI world, but copilot, which is basically AI autocomplete. So you type class foo and then you wait and just wait a second. And copilot sort of understands what you want to do and then gives you a first pass of what that class might look like. And it's basically amazing. When I'm on an airplane and stuff, I can barely work now because it's. I don't want to type all this stuff. I want this thing to be able to figure out what I want it to do and then write it for me.

A (24:34)

What phrase or sentence strikes fear in the heart of a software engineer?

B (24:38)

The site has been hacked Ooh, have.

A (24:41)

You seen that before?

B (24:43)

I have seen many sort of like bugs that you could potentially interpret as some kind of a little bit of a hack. But I've seen one case when I was working on, I was helping support one of my friends who had a crypto website and there was a long standing bug in the code that was years old, but somebody figured out how to turn it into a full scale hack and basically stole a ton of money from this person's users by basically taking over the screen and sort of like making it look like something else.

A (25:08)

And there's a real profit motive there. I mean, if you can break into the crypto sites.

B (25:12)

Yes. And then it was, the money was instantly gone and we traced it to, you know, Turkey. It was like it was instantly gone and not recoverable.

A (25:20)

What is an aspect of your work that you consistently savor?

B (25:25)

I love to reproduce a weird bug, a bug report from a user. So it's like we had a user who wrote in that all of a sudden our software was not generating nice pictures of birds anymore.

A (25:37)

So just to break in here, remember, Taylor's current startup is one that takes prompts from customers like, show me a pileated woodpecker pecking on a tree and the AI will spit out a polished looking video. So Taylor says their first reaction was that this bug report couldn't possibly be right.

B (25:54)

But then we went one step deeper. We like, you know, asked the person for more information and she's like, oh, well, actually, you know, if you do a pileated woodpecker or like a, you know, a female robin, like it might get that right, but it really won't get, you know, a coastal warbler or like a crested whatever, you know, those are just off the map. And I'm like, wow, she really knows a lot about this. So then you take that and you try to put the inputs in the right spot to figure out if the system's really messing something up. In the bird one, it was actually very easy to repro and it's extremely pleasing to be like, holy crap. Our system's actually really bad at birds. So sort of like connecting the dots of how the software is broken and then creating a path to fixing it.

A (26:34)

Is very fun because what does it represent for you? It's like all of a sudden something that seemed kind of random and noisy is actually diagnostic of something systemic that you understand. Is that the idea?

B (26:47)

Yeah, it's like a puzzle. So like every day at 5pm we get one person who emails us and says the button's Doing this, what's the deal? And then you figure out, oh, there's actually this job that runs every day in the background and it temporarily puts pressure on the database. And thus this other query that normally works totally great is failing and timing out for this user who happens to be in Istanbul. And like, isn't that cool?

A (27:13)

This is my very last question, I promise. This is from a listener. This is sort of a weird meta question. What question do you wish people would ask when they first hear what you do for a living?

B (27:24)

I think the problem is that most people don't know where to start. Like I recently moved to the Midwest from San Francisco. In San Francisco, when you meet a random person, the likelihood that they're a data engineer at whatever company is like very high. So they will know exactly what you're talking about most of the time here I think people don't know what that even means. So I say like, oh, I have a software startup, we're building an app where people can build videos and they can type in spooky story at night and out comes a video. I think people are like, what?

A (27:51)

That seems very clear to me. I mean you didn't use a lot of jargon in that pitch. I don't know.

B (27:55)

I know, but I think the job of building it is so weird. I think most people, when you think of consumer software, I also have a photo sharing app and I think when people see that app they think like, well there must be a company about it and there's probably a couple dozen people who work on it or whatever. And I'm not going to get one of those people when I send an email to this email address certainly. And it turns out it's like just me and my co founder getting an angry email from somebody and they didn't realize that it's like, well there's three of us right now and like yeah, of course I care about your bird problem.

A (28:24)

That that is so wild that you're like the founder and you know, world class engineer and like tech support.

B (28:31)

Yeah, we're right in the weeds. But that's how you find out when everything's on fire and that you need to fix your bird model.

A (28:42)

Taylor Hughes is a software engineer. Right now he's the co founder and CTO at the startup Hypernatural AI. You can check it out at Hypernatural AI. I keep thinking about the idea of tech debt. The choice you make to put off certain foundational work for the sake of short term benefit. Taylor highlighted the natural tension that exists between product managers who want to get the new feature out there right now, and software engineers who want to slow down and perfect the current system. And obviously it's not like one side is right and the other is wrong. You've got to have a balance, and that balance might well vary by what you're doing. So if you're creating video games, then holding out for code perfection is probably not a good choice. But if you're building airplanes or heart valves, we'd really like to see those perfectionists in charge. This same tension pops up in so many different places. Film directors want that perfect lighting in the perfect scene, and producers want the movie done on time and on schedule. Or think about architects and builders. The architect wants that elegant curve in the wall, but the builder says it'll take three extra months and cost five times as much. I don't want to exaggerate the distinction here. I mean, after all, Taylor has to operate on both sides of the line. As we heard, he has to hammer out that good enough code for today while also tracking the imperfections for a future improvement tomorrow, tracing mysterious errors to their source, jousting with product managers, wrestling with thickets of interconnected code, and rolling out new features without breaking the old ones. Folks, that's what it's like to be a software engineer. Thank you to Jake Knapp for suggesting we talk with Taylor. And a shout out to recent Apple podcast reviewers Tim Taylor, 379-ABCD-157 and Dexistential. This episode was produced by Matt Purdy. I'm Dan Heath. See you next.