Stop being skeptical about AI for development with Charity Majors

Summary

---

Episode Overview

Podcast: The Pragmatic Engineer
Host: Gergely Orosz
Guest: Charity Majors (Co-founder & CTO, Honeycomb)
Title: Stop being skeptical about AI for development with Charity Majors
Date: August 12, 2026

This episode dives deep into the evolving relationship between software engineering and artificial intelligence. Charity Majors, a renowned engineering leader, reflects on her changing stance toward AI in development, the growing pains and opportunities faced in the industry, and the cultural shifts required to adapt. The conversation covers AI's practical impact on coding, the changing definitions of productivity and quality, reliability concerns, career advice for engineers and managers, and lessons from ops and QA for building and validating AI-generated software.

---

Key Discussion Points & Insights

Charity’s Background and Perspective Shift on AI

• Background: Charity recounts her career starting at Linden Lab (Second Life), lessons from Parse’s acquisition by Facebook, and the origin of Honeycomb inspired by internal Facebook tools like Scuba.

  • "Most acquisitions fail. Most were terrible. This one failed. They shut it down. But ultimately, I'm very grateful… it led to Honeycomb." (Charity, 02:47)
  • "I would be so much less powerful as an engineer without this [Scuba]." (Charity, 04:25)

• AI Attitude Evolution: From skepticism (“legit to be skeptical”) to conviction that AI’s influence is inevitable in dev workflows.

  • "Spoiler alert: the question is not if we will stop reading code written by AI, but when." (Charity, 00:00)
  • "In retrospect, you could see it coming sooner… it's the harnesses, the tooling." (Charity, 08:42)

---

Measuring Developer Productivity in the AI Age

• Old vs. New: The perennial question of measuring individual developer output is revisited; both host and Charity argue that team output matters far more than individual metrics.
  • "Teams are still what matter... AI is forcing us all to ask ourselves, what does good look like?" (Charity, 06:06)
  • "We all jumped so fast to speed. ...That is a very immature description of what better is." (Charity, 07:11)

---

Shipping Code You Haven’t Read: Inevitable and Manageable?

Theme: The industry is heading toward shipping code not directly read by humans; focus must shift to the conditions and guardrails that make this safe.

• "Talk about what it would take for you to be comfortable shipping code without you reading it... Because that is engineering." (Charity, 11:14)
• Lessons from Ops & QA:
  • "OPS and QA have always been about 'does it work', validating correctness," (Charity, 19:07)
  • We already trust unreviewed code in practice: "You probably haven’t read all the code in your codebase either." (Host, 00:00)

---

The Allure and Risks of Speed: Quality and Reliability Concerns

• Recent AI adoption is bringing reliability headaches:

  • "People are shipping faster than ever, but not necessarily shipping better." (Corey, 07:24)
  • "We're not talking about quality. ...Do customers really want the buttons on their app to move around all the time? I don't think they do." (Charity, 08:06)

• On reliability actually getting worse:

  • "Incidents are up everywhere. Reliability has gone down and it hasn't recovered yet." (Charity, 39:27)
  • "Inside of Meta... there has been a flurry of sev zeros. ... It's all of us." (Corey, 39:50)

---

Trust, Testing, and the New Validation Paradigm

• Old way: Reliance on code review and human understanding

• AI/Non-determinism: Requires new trust mechanisms like tests, evals, harnesses, QA-style behavioral/conformance tests.

  • "If you're debiting from this trust account in the creation of the code, it has to get built up somewhere else." (Charity, 27:25)

• Code review is overloaded:

  • "Code review means so many things to so many people... Do we want this in our product? That is what humans are good at. Validation, in my book, is at the very bottom of the list." (Charity, 22:17 / 23:59)

---

The Two AI Camps

• AI-Pilled (Enthusiasts): Embrace the speed, innovation, not worried about gaps.
• Skeptics (Operators/On-call): See increased slop, incidents, and system degradation.
  • "The people who are on-call for these systems, they're seeing more incidents, carelessness caused by AI, since that group started to use more AI, our systems are getting way worse." (Corey & Charity, 39:00)
• Neither side is "making it up"—real problems and real progress co-exist.
  • "We’ve never seen technological change this fast. ... Both sides feel like the tiny minority, but both are right." (Charity, 37:27)

---

Observability in the Age of AI

• Observability's essential role grows with AI-generated and agentic systems due to loss of determinism and the need for validation.
• Lines of code are no longer the ideal artifact to review—observability, automated tests, and telemetry are increasingly crucial.
  • "Telemetry should be a product decision.... For your code, rich data goes up combinatorially." (Charity, 53:19)
• New book: "Observability Engineering" second edition, doubling in size, focuses on deterministic/non-deterministic systems, organizational practices, and governance. (Charity, 61:15; 63:24)

---

Engineering Leadership & Career Advice

• On Management:

  • "Middle management is deeply essential... but everyone should get AI on their resume. Go back to being IC for a while if you need to." (Charity, 69:51/72:37)
  • "Management's role is sense making and context giving... we don't live in a world where you just assign tasks." (Charity, 70:03)

• For Junior Engineers:

  • "The kids are gonna be okay. We just have to give them a shot." (Charity, 75:44)
  • Internships are a great opportunity even in uncertain times.

• On AI Fatigue:

  • Multiple varieties: hype, doom-trolling, family/cultural fear, slop overflow, and decision fatigue. Advice: Take agency, run team experiments, and take control back.
    • "Take control back, try something, propose something. Better yet, don't even tell me, come and tell me what worked afterwards..." (Charity, 79:44)

---

Notable Quotes & Memorable Moments

• On rewriting code, “code as cash,” and the future:
  • "If you could generate 10,000 variants of a function faster than you could write one, we're going to need a lot of evals, a lot of tests... I don't know how far we can go, but we can go farther than we are now." (Charity, 14:06)
• On being human in a world of agents:
  • "I'm so excited about leaning into the parts of being human together… I crave this more full… I'm so glad you're here." (Charity, 47:59)
• On business realities for tech leaders:
  • "If we don't do it ourselves…someone will come and do it to us." (Charity, 68:09)
  • "It’s not enough to be a good person. You need to be good at business." (Charity, 69:33)
• On attention and respecting colleagues:
  • "If it takes them longer to read it than it took you to make it, it’s probably slop. That’s really disrespectful, actually." (Charity, 34:05)
• Agency in the face of anxiety:
  • "Anxiety and excitement are psychologically almost the same, but the difference between them is agency." (Host, 75:00)

---

Timestamps of Important Segments

• Charity’s career origins and Honeycomb’s inspiration: 02:05–04:55
• Individual productivity and team/AI influence: 05:38–07:24
• History of skepticism, AI’s advance in development: 08:14–11:50
• Shipping unread code / trusting AI: 11:14–14:53
• Reliability going down, AI’s costs and wins for teams: 39:00–43:23
• Observability, measuring the real state of software: 53:17–57:33
• AI camps, operator vs. enthusiast, and system degradation: 36:48–43:23
• Career advice for leaders and managers: 69:51, 72:37, 74:12
• On junior engineers’ future: 75:35–77:05
• AI fatigue and taking back agency: 77:06–80:44

---

Recommended Books

• Catastrophe Ethics by Travis Rieder — On ethics in complex, interconnected dilemmas. (Charity, 80:52)
• More Everything Forever by Adam Becker — Critique of tech utopianism, AI religion, and singularity thinking. (Charity, 82:02)

---

Closing Thoughts

Charity emphasizes that while AI brings incredible capabilities, it also exacerbates issues like reliability and cognitive overload. The best path forward is honest reporting of both wins and costs, refocusing on the human and collaborative side of development, and adapting organizational and technical practices for this new paradigm.

As AI redefines what it means to build and trust software—not if but when—software engineers, leaders, and organizations must rise to meet the challenge of building systems and careers in this new reality.

---

Transcript

[Host]

Why are there firmly two camps within software engineering when it comes to AI Those hating its effects and those who are AI pilled. Charity Majors emphasizes both camps and thinks they are talking alongside one another. Charity is a co founder and CTO of Honeycomb, previously worked at Facebook and Parse, and is one of my favorite voices in engineering. Today we discuss what it would take for us engineers to ship code we have never read, and why this is more of a when question, not an if question. Why reliability is quietly getting worse across the industry, and why it will take some time to recover. Career advice in this age of AI, why middle managers should consider going back to being ic and why junior engineers will be okay if you want to hear from someone who was skeptical about AI in 2025 but has changed her mind based on the evidence, this episode is for you. In today's episode, Charity will say spoiler alert that the question is not if we will stop reading code written by AI, but when. And we should take lessons from OPS and QA on how they prove that software that others wrote works in producers. And she's got a very good point. As any OPS engineer or SRA will tell you, that's how software has always been written by unreliable agents from their point of view, that is Software engineers like me, your colleagues, or you. And let's face it, you probably haven't read all the code in your code base either. This is where I need to mention our presenting sponsor, Antistasys. Antistysis verifies software written by unreliable agents. It runs your whole system in a hostile simulation and roots out the box for you. It does this by using an approach called Deterministic simulation testing, or dst. Antithesis is turbocharger testing by running your whole system under aggressive fault injection. Imagine Antistysis's hundreds or thousands of versions of the Mario game, running each instance aggressively, trying to break the game with increasingly weird input combinations. If it finds a breakage, this is where the determinism comes in. Instead of you having to try to reproduce a tricky bug you saw in production, Antithesis can provide you with a perfect deterministic replay of anything it finds every time. With Antistasis, you can specify properties at the whole system level, and Antistasis will actively try to disprove them. So you can be confident that if your system holds up in Antistasys, it will hold up in production. Head over to antistasys.compragmatic to learn more.

[Corey Quinn]

Charity it's so nice to do this in person.

[Charity Majors]

You're in my City. This is amazing.

[Corey Quinn]

So today I wanted to kick off with AI, but before we kick off with AI, I just want to make it kind of clear for people who don't know you that, you know, you're not an AI hater or an AI lover.

[Host]

You.

[Corey Quinn]

You actually built a lot of cool stuff pre AI, right? Starting at. We just saw Linden Labs. Was that your first job?

[Charity Majors]

My first job at Linden Lab. Right across the street.

[Corey Quinn]

Right across the street. We were just talking about that. So you were building Second Life?

[Charity Majors]

Yeah, we were building Second Life, yeah.

[Corey Quinn]

And then from there on, one of the big kits was Parse, the developer tool which was beloved by developers. Backend for all, best backend for mobile services. I used to use it. And then what happened? Facebook bought you?

[Charity Majors]

Facebook bought it? Yeah. It was my first great lesson in most acquisitions fail. Most were terrible. This one failed. They shut it down. But ultimately, I'm very grateful to have had the experience, because if it wasn't for that, I've always been a startup kid, and so nobody knew my name. And it wasn't until I was leaving Facebook that investors were like, oh, would you like some money? And that's how we started Honeycomb.

[Corey Quinn]

And then you saw stuff at Facebook, right? It inspired you that.

[Charity Majors]

Yeah, yeah. Facebook, There's a tool called Scuba. And so we were in a weird position. We were building on aws, Ruby on Rails, all this stuff. And then we got to use the internal Facebook tools, and Facebook had this tool called Scuba. And it was. We were experiencing hockey stick growth. It was just like we had over a million mobile apps hosted on Parse by the time I left. And every single week, a new one would break. It would hit the top 10 on iTunes or something out of nowhere. It would just be like, ah. And these apps, needle in the haystack, you know, and it went from. Would take hours or weeks. We have to get lucky. We finally find. Because it's not a. It might be one app that's spamming your logs, but that might not be the reason. They might all be backed up behind the reason. You know, we started getting our data sets into Scuba and finding them. It just went from being a really hard engineering problem with a lot of luck to just being like a port problem. Click, click, click. Oh, there it is. And it was just mind blown like you just. That was a huge problem for our entire existence.

[Corey Quinn]

And then it was solved with Scuba. And then when you started Honeycomb. So was this a bit of inspiration that you wanted to build something that feels like Scuba did?

[Charity Majors]

Yeah. I just, the idea of, I was planning to go be an engineering manager, an engineer at Slack or Stripe or something, and I was just like, oof, I would be so much less powerful as an engineer without this. And so, you know, the grand plan in the beginning, I'm just like, well, all startups fail. So, you know, we'll fail, but I'll go sit in a corner and write go code for a year or two and then, then I'll open source it and I can take it with me wherever I go.

[Corey Quinn]

That's how Honeycomb started.

[Charity Majors]

That's how Honeycomb started.

[Corey Quinn]

And we'll get back to like observability or Honeycomb. But before we do, now with AI, you know, it's changing everything. But I kind of had a bit of a blast from the past, which is when of the first places we connected was in 2020. So almost five years ago or so, someone submitted a question to both my blog and your blog. And it was, the question was like, can you measure individual developer productivity? Now, I wrote an answer and you wrote an answer and I wanted to ask you. That was five years ago, no AI, no nothing. Today someone shoots you a question saying, hey, charity, can you measure one of an engineer's individual productivity? You know, they're using AI tools and all this stuff. What would you tell them?

[Charity Majors]

I would tell. God, I don't even remember what I said. I remember that blog post.

[Corey Quinn]

But we both agreed, by the way, that it was, it was that you can measure some dimensions and they're not going to give you the full thing. And they will, for example, not tell you how a team is doing if someone is, is actually really a key part of the team and that as long as you measure individual things. We both agreed that you need to be in the details to know. And as a good manager or a good team lead, you will know.

[Charity Majors]

You will know, but you have to have data to back it up. It's like color and a painting on the wall. And is it Goodhart's Law? Yes, it's Goodhart's Law. So like never go well. It's this thing that matters, right? You need to actually understand, but you need it to not just be your opinion that was tossed off because you, you have an opinion about some per. You know, we're, we're all, we have biases. We, we are selective, you know, you know, need to look at the picture. I also believe that, you know, there's been this whole push towards individual output, but teams are still what matter. Team out. And honestly if there's one thing that I am encouraged and excited about with the AI movement, I think it's forcing us all to ask ourselves early and often, what does good look like? What does good mean? What does productivity mean? What would better look like? What would great look like? You know, and these questions are hard. I think it's telling that we all jumped so fast to speed.

[Host]

Yep.

[Charity Majors]

Oh, fast. We can do it fast. Let's do the same thing faster. You know, just boom. And I've come to feel like that is a very immature description of what better is.

[Corey Quinn]

Yeah, just. Just today, I saw the Anthropic team posted a podcast with Spotify's head of engineering or VP of engineering. I'm not sure which one in which they talk that. Wow. Spotify with cloud code, they're shipping 4,500 changes per day, per week. I'm not sure which one, but they talked about speed. And I was kind of thinking, like, my experience has been different because I struggled to publish any. Like, some of my episodes did not go on Spotify because it was down.

[Charity Majors]

Yeah.

[Corey Quinn]

And yeah, they're talking about speed, but we're not talking about quality. We're not talking about more functionality, better functionality, or just things that people want. And in a comment, some people are asking like, okay, so what exactly does that mean that they're shipping more frequently?

[Charity Majors]

Yeah. Do customers really want the buttons on their app to move around all the time? I don't think they do.

[Corey Quinn]

Yeah. It's an interesting one.

[Charity Majors]

It's the easiest thing to measure.

[Corey Quinn]

Let's jump back to last year. In 2025, you. You wrote a blog post right at the end of the year, looking back, saying that 2025 for AI was what 2010 was for the cloud. Can we talk about before we go into this year? But last year, how was your perspective? Of course, working at observability company, AI will give you lots of business as well. But you said it went mainstream, right? Last year.

[Charity Majors]

Yeah. In March of 2025, Hebert and I gave a keynote at Srecon. We gave the closing talk, and it's Fred and I standing in front of a. The. The term vibe coding had just been invented.

[Corey Quinn]

Oh, yes.

[Charity Majors]

And we were like, you guys should try vibe coding. Pause GROANS AUDIBLE GROANS Just like people laughing like, ha, ha, ha. Our big pitch was that people should learn AI, because you can complain better if you learn it, which is legit. I mean, I really need it. Um, but at the time, I think I still saw it as a really big feature or, like, A bigger than a programming language like the cloud, but not like generational, you know, not changing everything. And. And I think that was accurate. Um, for me, it was November of 2025 when they released Opus 4.5. But I actually wrote about this recently, a couple blog posts back, about how in retrospect, you could see it coming sooner. You could see. And it wasn't actually the models, it was the harnesses, it was all the tooling. And it was people starting to say that around July. They were like, this is coming faster than you think and this is what it's going to look like.

[Corey Quinn]

And it was people who were saying it the word ones who were playing with either a cloth coat or maybe pie or open coat. So the harnesses, you're right, they were

[Charity Majors]

getting better at the tooling. You know, it went from just being kind of a shell script that would try again to like, they built a lot of stuff around it. And then, you know, the opus thing kind of. It was a weird time at the beginning. It's been a weird time every. Every time for a long time. But the early months of this year, it felt like everyone around me was just trying it again and changing their mind. Everyone.

[Host]

Yeah.

[Corey Quinn]

But I think we were just talking about right before we started recording that both you and me respect people who do change their minds.

[Charity Majors]

Yes. And I don't think we were wrong to be skeptical that. That the first time. It's a pretty extraordinary claim that AI is going to write code about as well as the meeting software engineer can in, you know, for limited bounds of that.

[Corey Quinn]

Well, especially because if we look back at the history of software engineering, this claim has happened again and again.

[Charity Majors]

Yes.

[Corey Quinn]

You know, neural nets should have been doing something magical.

[Charity Majors]

There's a sticker in your pack that says we already have a programming language that lets you COBOL as the punchline. So I don't think we were wrong to be skeptical.

[Corey Quinn]

And also, don't forget no code and low code.

[Charity Majors]

Oh, yeah.

[Corey Quinn]

I mean, we know it turned out to be a joke, but the promise was the same. And we were skeptical and we were right. And now we're skeptical again and we were wrong.

[Charity Majors]

What I was saying in that piece, though, was I think we were right to be skeptical that time. But now I see the same thing playing out with would you be willing to ship it some code that you didn't read? There's no point in arguing about if it will happen or when it will happen. Talk about what it would take. What would it take for you to be comfortable Shipping code without you reading it and understanding it. Because that is. That's engineering.

[Corey Quinn]

And it goes back to like, you know, my. My gut reflex would have been saying, oh, no, I would not do that, because I've been used to that. However, you're right, you know, it would if, If I could have a way to, for example, I could see the change. I could. I could tell that this was tested in like a harness or something. Same way where, for example, pre AI, if there was a. I had a team member who said, I vouch for this. And I've. I've hammered it and I trust that person. So, like, you're. You're right there. There's these things which are, of course, would never. But I thought that AI or something can do anything like that. But if it could, you're. And that's entering. Right.

[Charity Majors]

Or, for example, if you and the AI would. Would both do it in tandem for a few months and you would be like, you would get to. How much are they catching? How much am I catching? Is it about the same? Is it more, is it less? And you're training it and it's getting better, whether it takes five days or five years or whatever. I think it's pretty clear that directionally that's where we're going. And the other thing that I would say is, this is good for us. If you spent much time with the Phoenix architecture stuff that Chad Feller has

[Corey Quinn]

been writing, you have been quoting.

[Charity Majors]

Yeah, I've been quoting liberally. I should probably let you get to it in your own order, but I just feel like anyone who's ever done a painful rewrite should be on board with us.

[Corey Quinn]

Yeah, but here's a quote from Chaff Fowler. Immutable infrastructure, stateless services, containers, bluegren deployments, infrastructure as a code. These ideas all share a common premise. Never fix a running thing, replace it. AI pushes this premise beyond infrastructure and into application code itself. When rewriting is cheap, editing in place becomes risky, mutation accumulates, entropy, replacements, resets it.

[Charity Majors]

Yes, code is cash.

[Corey Quinn]

This is a very interesting idea because you've compared. Chaff compared and you've also, of course, shared this, that when we look at how infrastructure changed before, you know, like specifically a server, you need to be configured. And I think we call it like pets versus servers having pets versus pets. And at some point we stopped configuring individually. We stop, like fixing individual machines. We just like throw it away and have a new thing. And with code, we've always been used to the history of the profession. You know, 60 plus years or maybe a bit longer is that we edit code. And are you thinking this might because

[Charity Majors]

of the economics of it? I mean if you, if you think about it, you could generate 10,000 variants of a function faster then you could write at once. And so when you start thinking about it that way it's like, well, okay, we're going to need a lot of evals, we're going to need a lot of tests. But the generation is so cheap that it really, I think it forces us in that direction. And I think that the expensiveness of writing code and maintaining code and the expense of software has always been bound up in its maintenance and those lines of code. The reason that we trust something is because we've been using it because we

[Host]

know

[Charity Majors]

there's this deep thing about production. It's like, well, it's trusted. We know. And I know that as well as anyone. And I will also say this, anyone who's ever done a hard database migration should have some real humility about our ability to extrapolate those contracts and store them. Like, I am not one of the people who's like, this is good, we're going to generate all code. I don't know how much code. I believe that we can go some distance in that direction and it will be good for us. I don't know how far we can go. I believe we can go farther than we are now. I just can't. The last project I did at Parse, so we had spent like 6 months writing the original Ruby on Rails API.

[Host]

Yep.

[Charity Majors]

Spent 2 years rewriting it in Golang.

[Corey Quinn]

Wow.

[Charity Majors]

Yeah, it was, it was, it was.

[Corey Quinn]

And what was was it two years? Because new stuff being kept being added that you need to some extent.

[Charity Majors]

And also Golang was a pretty immature language at the time. We had to write, you know, the MongoDB drivers and like the, all the other bunch of things. And also just like when you're writing in Ruby and MongoDB and JavaScript and everything is, you know, there's no type safety and it was just painful. Just, you know, in the strangler figs that they do, where you build the architecture outside the architecture and you literally find the contracts with your users by breaking them one after the other, like that just does not seem like the ideal artifact. We should be able to store them somewhere. We should be able to have architecture diagrams that we can review and discuss that generate that code to spec.

[Corey Quinn]

This is very interesting because some of these ideas, they've been around decades ago specifically, you know, if we had Grady Booch as a third person sitting here, the idea of like, hey, we can have architecture diagrams that translate to code. UML started there. I, I think Grady would disagree that like, he never wanted it to go there. But irrational software back in the 90s, they said, hey, you'll define UML. It generates code. It will be beautiful. Now it wasn't beautiful because I guess some complexity and turns out it generating code was still expensive and reviewing it. But I wonder if some of these ideas now might be just feasible.

[Charity Majors]

That's my hope. That's my hope. I mean, I, I, I'm just barely old enough that my first job, I was like 17 at university. I was a sysadmin. I remember when, you know, I didn't, I wasn't really aware of what was going on. I was just a kid. But yeah, I remember how stressful it was and how people were agonizing about how we'll never be able to get that information pack. And everyone adapted just fine. They, I think I read the systems that, you know, they built the systems that replaced them, but not as in replace them and work them out of my job. They built the systems and they spent their time writing code instead of like running updates by hand on every server in the closet.

[Corey Quinn]

And I guess this is an interesting one because clearly like the sysadmin role and profession has been, it doesn't exist today. It's kind of, let's just say it has been eliminated. However, the people who were sysadmins, they did understand the operating systems. They understood hardware.

[Charity Majors]

Yes.

[Corey Quinn]

They were in a really good position to adopt. And a lot of them just became either software engineers, product managers. I know someone who became a tech salesperson.

[Charity Majors]

Yeah, yeah.

[Corey Quinn]

So it's almost like, like, and I

[Charity Majors]

will hold that our generation of engineers still the best debuggers. I'm glad that people don't all have to learn about CPU and memory and all this stuff. But like, there's value in knowing that stuff. It comes in handy. I think there's some analogies there to the generation of code stuff.

[Corey Quinn]

Also, you know, you, you took a bunch of inspiration in your recent writing about both sysadmins, but also qa, and you wrote something interesting. You said lines of code are not the ideal artifact to review. And I'll quote a little bit from you. The tools to do this don't exist yet, but many of the ideas do exist. Most come from operations in qa, two domains that software engineering has historically been rather snobbish about. Should we lose Our relationship to QA and ops, where I feel we always put ourselves as software engineers here and OPS and QA somewhere and maybe time to eat some humble pie.

[Charity Majors]

Ops equals toil, right? Yeah, I think it's time. I mean, OPS and QA have always been more concerned with what is software engineering has always been much more concerned with how should it be.

[Corey Quinn]

So OPS and QA have always been more concerned about validating, about correctness, about does it work as expected? Does it work? To start with, does it work?

[Charity Majors]

Yeah, yeah. I mean, it's always weird to me just how much software engineers really seem to believe that the world exists in the repo. It doesn't. It's production. You know, the code has part of the information, some of it, it's very necessary. We need that. But like, I know some software engineers who. And okay, some, some, some places don't even let software engineers look at production just like how. I know a lot of people are very upset about AI and. But there are. The things that get me very excited, genuinely excited about AI, are that it is pushing the discipline in directions we have desperately needed to go for a very long time. Production is not what happens after development. It is a stage of development.

[Corey Quinn]

And you've been saying this consistently for pre AI. I'm just going to say it for those who don't, because I remember we've, I think we also bonded a little bit over. There was this thing called trending on Twitter when it was still Twitter and it was tech Twitter. Everyone was there who mattered and there was a trend going, it's Friday. Don't deploy some. Something that there was maybe a hashtag even, like, like, I'm not sure, don't deploy Friday. Or something like that. And the point was, it was, well, meaning it said, like, look, when you deploy, often there's an outage and on the weekend we don't want to do so there was saying every Friday, it went viral saying, don't deploy on Fridays. And you came in and you said, you know what, you should be able to deploy anytime without fear because you should be able to just know, you know, however that might be cicd. And then on top of this, you were like, no, like, you should actually just not even have a user acceptance testing environment at uat. You should just deploy to production, like intestine production. Right.

[Charity Majors]

As soon as you merge, it should go be going out. Like, you should have to stop the train to make your code not go into production as soon as you've merged. Absolutely.

[Corey Quinn]

And one more interesting thing Is you had a long train of thought about, like, AI and what it could be is one thing you said is our brains are not built for validation. Almost everyone I talked to, including Andres Heisberg, he said that look like it's very clear that code generation is cheap. We are generating more code. And the bottleneck for human engineers is for code review. And everyone's trying to figure out, how do we make code review easier, how do we build nicer tools? Uber has built amazing tools to, like, try to, like, surface important code reviews. But everyone's pushing, like, all right, let's do more code review. As an engineer, I'll be honest, like, I never liked doing a code review when there's very little to do and it's with someone I care about, I'll entertain it.

[Charity Majors]

It's more of a coaching opportunity then, right?

[Corey Quinn]

But, but, but especially as soon as there's an AI, it's kind of like, I don't know. I. I don't really care. Like, I'm just being honest here. Like, do you care when I don't.

[Charity Majors]

I've never. So the prob. One of the problems is that I think code review means so many things to so many people in so many places. And so a lot. There's a lot of projection going on. A lot of people are. If you say that you don't want code review, you're saying you don't want to talk to your co workers, you don't want to mentor juniors, you don't want to, you know, which is not true. We've just bundled so many things into

[Corey Quinn]

this, like, you know, it's like, hugely overloaded.

[Charity Majors]

Hugely overloaded. And some of those things are really good. Some of those things could be done better in other ways. You know, some of those things are very cultural, very specific. My friend David Poll, who I worked with at Parse, and he's now working at GitHub on pull requests.

[Corey Quinn]

Amazing. I love the Parse mafia.

[Charity Majors]

Yeah, exactly. He's like, to me, the code review is when we decide, do we want this in our product or not. I'm like, well, that is a. That's a great, great discussion. That is what humans are good at. We should talk about, is this mental model coherent? Should we add this? Should we not, like, love that architectures, you know, but, like, the code is not necessarily a great artifact for all of those. So should we be talking to people? Uh, yes. Is the code review the right form factor? Maybe, but I, I think that the emotional reaction, that's when people are getting to the like the validation in my book is at the very bottom of the list.

[Corey Quinn]

I. I'd like to like touch like stay here a bit more. Can you break out the. The parts. Because it feels to me code reviews are overloaded but the parts of code review or. Or the things that you have seen are good things and maybe we don't need to as code review and the things that are just like just have never been that good and yeah, maybe we just need to throw it away. Yeah.

[Charity Majors]

I mean I think do we want this in our product? Is that is great. I mean ideally you'd talk about that before you write the code for it but you know whatever and you know is this API design. You know those are great conversations reading for syntax and bugs and that sort of thing. It's not evil but it feels like it could be. It's a. It's a teaching opportunity if that's the best teaching opportunity you have. And I guess some folks at some point maybe need them. But it doesn't feel high. Is it like a great use of anyone's.

[Corey Quinn]

It feels the only time where it's useful is if someone joins a team and initially it can be a little bit of feedback.

[Charity Majors]

Yeah.

[Corey Quinn]

Especially when there's like nothing is written down. There's no guidance, there's no linting rules that would give you that.

[Charity Majors]

Well see that's again yes, we can fill in the cracks. If we haven't built the guardrails, we can fill in the cracks all kind of ways with. With our own time. But there are so many things I think that we never think to extract out of the process of building and validating software. We rely on us. So I am a huge fan of Intercom now Fin their engineering Org and I have been forever. Like I noticed their CTO decade ago had this saying shipping is your company's heartbeat. And I love that they ship a ruby monolith like 10, 15 minutes hundreds of times a day. That is not trivial. It's not trivial thing to do. Right. So they're kind of a high watermark in my mind right now for teams that were founded pre AI have a lot of engineering discipline who have become AI native and they wrote a great post about how they do PRs that are AI validated and the bar for them is very high. It's like they have all the wisdom of their most senior engineers looking at every single diff and that is fantastic. Which means that you don't have to worry about remembering and looking and nitpicking and all the things that we're not good at anyway, and they can talk about, is this the direction we want to go? Is this the right, is this the right path?

[Corey Quinn]

You've also written that non deterministic systems require more engineering discipline, not less. So like, what is the thing about these non deterministic system specific AI, Right, we're talking about AI, let's just name it. That is, we see that AI does amplify both discipline and lack of discipline. Why do we need more? And when you say discipline, what specifics

[Host]

are we talking about?

[Charity Majors]

Well, I mean, tests and evals for one thing, right? Like if, if we're treating the code like a trusted artifact and we're, you know, trying to predict everything with our human brains and everything, then we're writing the tests that we can predict that it might break. You know, and then anytime it, anytime the system breaks, we like try and write a test for that. But that is not an especially high bar. And so I think the sort of the behavioral tests or the, I don't remember the word, it starts with C. But the QA folks have these suite of tests where it captures.

[Corey Quinn]

There's also smoke tests.

[Charity Majors]

Yeah, there's so many.

[Corey Quinn]

There can be like performance tests, there can be load tests, there can be, yeah, there can be like just kind of fuss testing as well.

[Charity Majors]

Something that's like, if, okay, if I'm not going to read this code, how do I know it's going to perform within boundaries of the last code that I generated? That is conformance testing.

[Corey Quinn]

That's performance testing.

[Charity Majors]

Just as important for lots of workloads as, you know, absolute performance. Is it just not changing too much? And so we, I think we're going to need the trust test to go somewhere. Right? If you're debiting from this trust account in the creation of the code, it has to get built up somewhere else. And I feel like one of the things that I'm really excited about in the coming months is just I actually really like thinking about it less as AI and more as deterministic and non deterministic systems that have to play nicely together because determinism is not going anywhere. It's incredibly valuable. And we have to learn to make AI kind of boring. You know, it's a non deterministic tool, which means that it is all over the place. But it's so valuable. But it's all over the place. We have to learn how to give it carved pathways and places where we kind of corral it, where we use it in the way that it, it's a superpower and not in the way that like erodes our foundations.

[Corey Quinn]

This is interesting as Martin Fowler a year ago when he was on the podcast, the thing that he talked about is how the biggest change with AI is the non determinism. And when we think back in the history of software it's always been deterministic. Same for neural nets. But that was most of us software engineers didn't really touch too much of it because it just wasn't that useful for us. But we've been used to that. When we programmed it, you know, it just happened the same way. Unit tests were easy because you just run them once, you don't really run them twice because why would you. And I wonder if this is.

[Host]

We need to just realize how big

[Corey Quinn]

of a deal this change is and that the, any business that employs us like you know, they, they want software that works the same way. We just had a recent post on Hacker News. There's this ATS Application Tracking System scoring system that HackerRank outsourced which scores your resume. And so software engineer just like and, and you can run it locally, it's open sourced, you can use a local model. I think they recommend Gemma, Google's small model. And when you run it like a hundred times it will score the same resume anywhere from like 66 point to 99 points. And typically most companies have 85 set as the bar. And you're like hang on. So we've turned what is what they were advertising as a tool to help your recruitment. We just, we just prove that it's just a coin flip. Like that's bad.

[Charity Majors]

Yeah. And we have to be able to say that it's bad. AI is not the right tool for every use case. You know and I think, I think every company is going through this in microcosm. And something I was saying to folks just earlier today, we are, we've been doing these series of conversations on our AI norms and values and it was like a year ago. I, I don't trust us like a year ago if we were like yes, we should use AI. No we should, we didn't, we didn't know enough. We've gone on such a journey over the past year and we know so much more now. The like if one of my co workers is like AI is the wrong tool for this job, I'm like I trust you. You know, you gotta get worse before you can get better.

[Corey Quinn]

So tell me about where you are right now with your how inside of honeekompy are thinking about AI. How you're thinking about how to think about AI and what, what, what values you came up with that works right now for you?

[Charity Majors]

Yeah. It starts with just acknowledging that the bar has gone up for all of us. That's what happens when we get powerful new tools.

[Corey Quinn]

Has the bar gone up or has the, the, you know, the floor gone up?

[Charity Majors]

That is a great question. Maybe, yes. Maybe. Yeah. I don't know. We're. We're definitely in a sort of wandering in the wilderness phase, but you can't not wander or you will be left behind. You know, we acknowledge that the bar is going up for all of this and that the only viable way to define that bar is better outcomes. And asking ourselves like, is this good? Is this better? What does good look like? Another. Another thing that they point out is just there is no human in the loop. You own the loop. The loop is yours. The loop is mine. It would not exist if it was not for me. So I am the owner, right? There's no oh, Claude said this, so. No, no, no. It's your work. You own it.

[Host]

Charity just talked about owning the loop. Owning the loop also means controlling what every agent inside of that loop is allowed to do. Which brings us to our season sponsor Work os. Today, agents are increasingly able to act on their own and the old AUTH model was never designed for that. Who is this agent? What's it allowed to touch on whose behalf? You really don't want to get answers to these questions.

[Corey Quinn]

Wrong.

[Host]

WorkOS is built exactly to solve this problem. WorkOS is fine grade authorization FGA designed for how agents actually operate. Plus SSO and SCIM and not just user auth. With agents bolted on after the fastest growing AI companies. Anthropic OpenAI Cursor Perplexity already trust WorkOS. Check it [e-mail] I also want to talk about Buildkite, the CI orchestration platform trusted by Cursor, OpenAI, Anthropic, Nvidia, Uber, Canva, and more. Charity talked about owning the loop, but here's a thanks to AI. Your agents are writing a lot more code to trust this code. Every change that an agent makes still has to be built, tested and proven safe before it ships. So Obviously you need CI more than ever. But when agents are pushing 5, 10 or 50 times the commit volume to your pipelines, faster CI runners won't be enough to keep up with it. Shaving 30 seconds off a single build is meaningless when the queue is 100 jobs to what you really want is a CI system that gets faster as the volume grows and CI that offers instant parallelization to give you unlimited concurrency and to intelligently route changes at runtime. This is what buildkite does and why global software leaders at every level continue to rely on it. The same architecture that observed the scale of Shopify and Uber a decade ago now runs about 1.4 billion job minutes a week across Cursor, Meta, Reddit and Snowflake. While the rest of the CI world are crackling under the weight or re architecting their platform, buildkite continues to reliably grow. Agents run on your infrastructure or on buildkite. Any cloud, any chip, your secrets, your scale, every artifact and log is captured. So when something fails, either you or your agents have immediate insight for why. As you're entering the context you give to your agents, think about how you'll verify what they hand back. If your system is buckling under the increased volume, head to buildkite.compragmatic 30 day all access trial, no credit card and an actual human NGO on standby. His name's Ola and he's very helpful. And with this, let's get back to charity and communication norms with AI.

[Charity Majors]

I think there was this frenzy of oh my God, I can do this, oh my God, it's so cool. And I, I, I know you have also become very weary of this slot. I just don't even read it anymore. As soon as I can tell, as

[Corey Quinn]

soon as you know, I recognize this might have been AI, it's like trash.

[Charity Majors]

Here's a baseline. You cannot send anyone something you haven't read. And in fact, if it would take them longer to read it than it took you to make it, it's probably slop. That's really disrespectful actually. And I think like, just like asking someone to like you're asking, anytime I give you something, I'm asking for your time and attention. And if I'm giving you something that I don't even know what's in it and I and I'm putting it on you, it costs you instead of me. That is not good. I also think that even before that it's like I've noticed as I start working on these norms and values, I'm noticing myself as I start to ask someone a question without trying to look up the answer, ooh, I shouldn't do that. Or if I'm giving someone something that I kind of generated and I'm like, ooh, you know, it's so part of it is just self awareness.

[Corey Quinn]

It's interesting because everything you talked about, it reminds me of when a new joiner would join a team. A junior engineer, a new grad. Either they had emotional intelligence or they picked up on really quickly that. For example, you go and ask a senior of their time once you put in a little bit of work and you start to respect their time as well. And obviously it doesn't start like that. We don't want them, but there's this balance. And I almost feel it's the same thing. We're like, look, respect your colleagues, respect fellow humans. If you are communicating with them, make sure that you're not wasting their attention. Because now I guess attention is where we're kind of running low. Like we have all of these, all of these, like a bunch of people have a bunch of agents doing. But the point is that's kind of the currency. And as long as you respect that, it doesn't matter. Like, I think we're not talking about don't use AI for this or that. Like you use it as much as you want or make yourself more efficient. Just don't degrade because it really degrades those personal skills, right?

[Charity Majors]

You can use AI as a shortcut to help you not have to think too much, and you can use AI to help you think more deeply and more rigorously. And both of those use cases have their place. But when it comes to your core job function, we primarily want the second one, right? And especially if you're involving someone else and you're asking them to review or you know, and this is not absolutist, like there are people who, English is a second language and they use it. People who like neurodivergent in these. And that is again, that is still being respectful. You know, it's so it's not like, like you said, it's not no AI, but it's like make reasonable asks of each other and, and you know, we don't need to reinvent a new bar for quality or respect because we have great bars already for quality and respect. We just need to apply. For a while there, I think that there was a bit of, oh my God, this is so cool. Do you see what this cool thing can do? And I think we're all just like

[Corey Quinn]

so over the reason. I really respect that you came from the CIS dev background. You also, you're very involved in sre. These are all folks who have been pretty skeptical of AI. And you mentioned how you're seeing two camps, two very clear camps. There's like kind of the AI Pill folks who get it, and then the people who seem to, like, they just hate AI. And you said that you're not seeing these two camps have any sort of way to go between any feedback loop. Can we talk about what you're seeing? And, like, maybe, you know, like, where you see some of these camps forming.

[Charity Majors]

See, the problem is that neither side is making it up. Like, they are seeing really scary trends. They're seeing. They're grappling with real hard problems that are getting worse. You know, and on the enthusiast side, it's like they're acutely conscious that it's. It's a bit of a race and that we need to push ourselves out of our comfort zone. And they see other companies moving faster, catching up, leapfrogging. They're really worried about, you know, we're falling behind. And. And the first thing, I don't want to make it sound like false equivalence, because while there are elements of this that are true, lots of. I think every company is more one or more the other. But, like, they're not wrong. They're not wrong. These. We've never seen technological change this fast. We're on the inside of an exponential curve, which is very rare. And it never usually lasts that long, but it's still happening. You know, things are happening that shock us, and we would be wise to prepare for them. So, like, that's real. That's real. And. And these folks are usually, at most companies, usually they are the small minority, and they are constantly filling out, man. One of the things that's ironic, though, is that both of these sides feel like they are the tiny minority and they're out mad and they're being suppressed and they are standing up for what is truth and valor in the face of the big AI folks or the big skeptics, but the other side. So. And this often starts to come down to the group that is on call and the group that is not. Ooh.

[Corey Quinn]

Yep.

[Charity Majors]

Because the people who. The buck stops with them, they are seeing melting mental models. They're seeing slop. They're seeing all their hard work just dissolve, and they're see. And they don't see any end in sight.

[Corey Quinn]

So just to be clear, we're seeing that the people who are on call for a lot of these systems, they're seeing more incidents. They're seeing carelessness being caused by it. They're actually seeing that since that group started to use more AI, our systems are getting way worse.

[Charity Majors]

Way worse. Yeah. And that's very real. I'm not making it Up.

[Corey Quinn]

No, no, no. Actually, I was just talking to someone inside of Meta. There's been this big Dr. People have your post. So not just my post since then. I haven't written about this since and I'm not sure when this podcast came out, I might have not talked about it. Is inside of Meta. They track sev zeros, which is the high.

[Charity Majors]

I remember.

[Corey Quinn]

Well, you remember sev zeros. There has been a flurry of sev zeros. So many of them. And you cannot hide. Like, this is, you know, Meta. Like this is black or white. And the past about two months, it's been crazy. And just so it happens. It's happening inside of Instagram, it's happening inside of WhatsApp, where the trust and safety, basically the reliability folks have been axed, removed. So it's impossible to deny the connection as well. Of course, it's not a direct one, but again. And each one has as a postmortem. But Meta has not had this badge for closer to a decade.

[Charity Majors]

Yeah, move fast and break things.

[Corey Quinn]

And you put two plus two together. And when I told this story at a conference, people came up to me and they said, I'm so glad you talked about this, because my company, different company, often VC funded or publicly traded, like, the same thing is happening. People are like whispering to me like,

[Host]

we are not meta, but the same thing is happening.

[Charity Majors]

Same thing is happening.

[Corey Quinn]

And you know what they all told me?

[Host]

They told me, I thought, it's just us, or I thought it's us.

[Corey Quinn]

And then my buddy who works at this other company, and suddenly it was like, oh, it's all of us.

[Charity Majors]

No, it's all of us. Yeah. No. What's the real thing? And the intercom folks, you know what I love about them is they publish the real gnarly stuff, right?

[Corey Quinn]

They don't color it out.

[Charity Majors]

They don't color it out. And they showed that for 18 months, reliability and code quality went down. And it had just started to possibly

[Corey Quinn]

be going back up, but it's. It's still not there. Where it was still not there.

[Charity Majors]

Where.

[Corey Quinn]

And they're very honest about.

[Charity Majors]

And they're honest about it finally. So this is the thing. Like, stop, like, spitting in my. And telling me that, you know, like, it's just. This is my thing. It's like, we need to hear the wins. We need to hear what's. We need to hear about what's possible. We need to hear what's exciting. But you gotta couple it with the costs, you gotta couple it with, is it worth it, you gotta couple it with, what are we doing, what is happening? And I feel like part of the reason both of these sides are getting so frustrated is because they're not those. They're not connecting at all. And so the people who are seeing really incredible. There are some really incredible things happening in software right now, like with rewrites and with, you know, automating away, like, real toil. And like, not a single person that I've talked to would give it up. Yeah, it's amazing that, like, they don't. They get so excited. Nobody wants to take it away. But half of the people are seeing the wins and they're not connecting it to the cost, which makes them think that, that their coworkers are just fuck nuts who are just like, oh, they just don't want to lose their jobs. They're just afraid of getting automated out of existence. They're just blah, blah, blah, blah, blah. Like, no, dude, you be on call and then see how you feel, you know, and. And there's a mirror effect kind of happening where the folks who are on call who are responsible for this stuff, they don't actually believe that these winds are real. They think they're all cooked because they're not hearing the quiet part said out loud that, yeah, we're seeing this win, but this is what it costs. We're still cleaning this up. We're still. And so that, that's my. That's my beg to everyone who loves Care Guys podcast and listens to this is tell the whole story, talk about the costs. We're all in it together.

[Corey Quinn]

Yeah, because you're right, like, this technology is not going anywhere. It will make a really big positive change at a bunch of places.

[Charity Majors]

It's here, but it's not magic.

[Corey Quinn]

It's not magic. And I think this is what you said in Make AI Boring Again, another great article of yours. What you said is, AI is just technology, just technology. And you were arguing that, let's just realize it's technology, it's a tool, and let's learn to use it. Well, now, one other thing you said, which is very interesting, is software will be the killer app with AI, which is very unique. Let's talk a little bit about that.

[Charity Majors]

Software is made of logic and language. AI is made of logic and language. And because of that, we can bake in guardrails, we can bake in checks, we can bake in validation that we. I don't know how we do that in other parts of our lives or other applications. And so it totally Makes sense to me that software is what AI is best at. I mean, you see like in the courts, they're starting to get lawsuits for. The court is suing lawyers who are submitting briefs that have hallucinated crap in them. How do you check for that? You know, with the same. We have structured data, we have, you know, a whole. And I just don't know how you account for that. In the same way.

[Corey Quinn]

It might also mean that whatever will work outside of the software industry for AI, it will be a subset of what will work in the second, basically, if we can do something with AI, if we can automate a process or something, you might be able to do it in other industries, but maybe not. But if we cannot do it, good luck, you will not be able to do it because we, we have the domain where you can validate stuff. We have, we have incredible training data on, on code that compiles, right?

[Charity Majors]

Yes.

[Corey Quinn]

Like in, in a place, bunch of places, you might have like training data. Like with, with magazines, you might have like low quality magazines or whatnot. Right. So you see what I mean?

[Charity Majors]

I mean, back to your point about humans, like their determinism, they like things to happen the same way.

[Corey Quinn]

And it's very interesting because as I think of it, you know, one of my businesses is writing. I write a newsletter that is, is. I like to think it's good and it's worth reading.

[Charity Majors]

It is.

[Corey Quinn]

And I would have said if you asked me, what is AI really good at? Now, obviously it's good at coding, but before that it was good at writing. It was like my mind was blown that it can actually control the language. When all the newer models come out, I do this test where I say, like, all right, like, you know, write an article in the style of the pragmatic engineer. And every single time I can tell it's AI generated because it's repetitive. It has this thing. So my point is, AI is actually not as good as writing prose, as it's a lot better in writing code.

[Charity Majors]

Way better at writing.

[Corey Quinn]

When I asked to write code, like, I often I'm like, yeah, this is something I could have written. Whereas when I asked it to write words, I'm like, I would have never written this. And it has training data on me. So who knows? This might prove that software is the best fit.

[Charity Majors]

I think it is. Software is a simplified version of language for a purpose. Yeah, I, you know, at first everybody was like trying to come up with ways to be more efficient and write with AI and everything. And I sunk a Lot of cycles into that. And I have decided not to think anymore, because writing is thinking on paper, and there's no shortcut for doing that. Thinking anything that I write, it's not content. You know, it's not content where it's just like, well, generate me a couple thousand words, which. I'm not shaming anyone who generates content, but that's not what I'm trying to do. I'm trying to think through hard and interesting problems and share them with people. And I don't think AI is the appropriate tool to use for that. I use it for structure. I. I'll be like, hey, read this and give me feedback and stuff, but not doing it.

[Corey Quinn]

Yeah. So I think we should not forget that as we improve our, you know, our. Our skills, our capability, our experience, our. Our thoughts, we do become more valuable. And I have this idea, and this might be a flawed idea, but I think it will be correct that five years from now, how will people be hired? Now, of course, we know the tools will be better and all that, but in the end, I think it'd be like this. Someone's sitting here, and I'm going to be interviewing with you. I'm going to be trying to get into your company, probably Honeycomb. Right. And we will be having a conversation, and you will judge me based on how I respond. And the more I have spent thinking and bettering myself, the more valuable I will be to you, because you will have all these candidates, and some of them will have outsourced or other things in AI, and they will have a blank because that thing is off. Guess who you will want to work with. Right?

[Charity Majors]

I am so excited about leaning into the parts of being human together. I don't like the feeling of chatting all day, back and forth between agents and people on Slack. Like, it feels way too similar. It's just gross. Honeycomb is a fully distributed company, which was never. We always wanted to have a hybrid model, but the office has not come back, and. And I. And I feel all kinds of ways about this because I love not leaving the house and. But at the same time, I. I crave this more full. Like, I'm so glad you're here. It's so nice to see you.

[Corey Quinn]

We were just talking how it. It is different. We. We've done a podcast remote and it was a decent one, but. But this is more enjoyable.

[Charity Majors]

Yes. And so part of what I hope we do is just remember that we're in charge of the machines they serve us, and this is still what matters.

[Corey Quinn]

Now I Want to pull back to. Back to something different to talk a bit more about OPS and DevOps and give one of your spicy stakes. So now that we have AI, we can actually just badmouth some of the other things or just be real. Let's talk about DevOps.

[Host]

Just.

[Corey Quinn]

Can we go back a little bit in time? You were there, why was it created? And in the end, there was this massive DevOps movement in the 2010s. Do you think it succeeded? Do you think it failed?

[Charity Majors]

So before DevOps we needed a DevOps because there was devs and Ops and there was the proverbial wall that code got thrown over. Right?

[Corey Quinn]

And Ops were the people who were in charge of the it. They deployed, they managed the servers, they set the Linux version, handcrafted Linux, you

[Charity Majors]

know, pluggable storage models and everything. That was always a bad idea because it's split brain. Half of you are writing the code and the other half are understanding it. I would argue that you can't really understand the code you write unless you're operating it. So, you know, the DevOps movement did a lot of good trying to knit back together that sort of original sin. And you know, around the time that I was a sysadmin, there was this big push. All right, OPS people learn to code. And great, I'm glad that happened. Everyone who works with computers should be writing code. I feel like the wave after that is a little less successful, which is like, okay, software engineers, time to learn to understand your code in production. But I also think that in my mind, 20 years of DevOps was really about one thing, trying to create one feedback loop that connected people writing code to that code in production. And it failed. I mean, it failed to this day, like, they're done, they're done by. They're two different domains. You know, there are some people who, I mean it's.

[Corey Quinn]

And I'll show you this diagram that you drew. We now added agents. We'll put it on the. So viewers can see it that this is your. I think it's a really nice drop of how there's no feedback loop like the OPS people or oftentimes we call it platform teams. They manage the infra layer, engineers deploy there.

[Charity Majors]

And so to be clear, I think that's actually good in fine and healthy. I think that there are separation of concerns where you can't expect anyone to do everything. And the nice separation of concern is, do I own? Am I responsible for the stability of the things that you put code on, or am I responsible for the code that I put on the thing. Right, that is a nice theme because you want the infrastructure to be stable, be like, to protect itself, to be resilient and all these things. And you want your code like to be oriented towards. Is every single user having a good experience? You can have one of those things be true and the other not be true. Like they're, they are decoupleable.

[Corey Quinn]

And, and actually this is like even the most modern companies, I, I often refer to Anthropic as this company which operates in a very different way to most companies. They're very successful despite doing a lot of different things. However, internally they have platform teams, they have the cloud platform teams and then they have applied AI, who, which is more of the kind of the feature teams, the integration and the two. I talked to both of them, they just have a very different outlook. They have a very different view on even basic stuff like will software engineers be obsolete? The people on the platform team were like, no, we're working really hard. And on the apply they're like, well,

[Host]

maybe it will happen.

[Charity Majors]

Yeah, that does not surprise me one tiny iota.

[Corey Quinn]

So this company Anthropic, that start with a blank page, they arrived at the same place.

[Charity Majors]

Yeah, yeah, no, I think it's the right separation of concern and I'm not trying to erase it, but I think that to be a good engineer you need fast feedback loops and this is part and parcel with the whole. Oh, the source of truth is the code. If that's where you live, if you live in the land of how it should theoretically work. No, and I think that with agents, they're breaking that, right? They're breaking that and they're forcing another thing on the observability trip is a lot of people, if you say like, what is observability? They'll be like, ah, well, there's three pillars, there's metrics, logs and traces. We talked about this last time. Metrics and logs, I would say are system exhaust. They're the exhaust pipe. They're. And, and they're never going away because every team runs a ton of third party software. They didn't write it, they don't own it, they just have to run it and it's outputting shit.

[Corey Quinn]

Yeah. And, and you want to.

[Charity Majors]

And you just gotta put a side,

[Corey Quinn]

you observe it, you see what'.

[Charity Majors]

Yeah, yeah, yeah, yeah.

[Corey Quinn]

And then you do stuff with it.

[Charity Majors]

Yeah. And you know, you should put it somewhere cheap. There's a ton of it. It's not super high value, but you definitely need it. Right. And you can't do anything about it. You just take it and put it somewhere. Then there's your code, there's your crown jewels, the code that makes you a company. And for that code, your telemetry should be a product decision. It should be. You store it once with all the connective tissue because the value of rich data goes up. Not linearly, not even exponentially. Combinatorially. If you have a wide event or a trace with 29 bits of data and you add a 30th, that 30th is more valuable than all the others. Multi like it is just so powerful and with non deterministic software, you know, right up front you can't predict what it's going to do. You have to like that is a product decision to capture that trace.

[Corey Quinn]

So let's talk specifically about modern observability and like companies that are, you know, like either building AI related code or just complicated code that they're generating, you know, in the old world, again like I'm just being, you know, like observerly101 back in a day. The way I would have written the code is you write the code and you think like, hmm, something funny might be going on here. Let me do a log or an info or a warn. And then I would also try to maybe if we're putting this in production, I realize that like okay, well I guess it's crashing and we don't have any logs there, so I guess it's some other part. Let me do, put a tool that will like log everything and I'll have a bunch of stuff. Now this is the old, the, the, the simplest way of thinking in kind of a modern business where I'm like, I know this is high value stuff. What are ways that I can go about that's actually maybe a bit like more practical than. Because I, I just will use Super

[Charity Majors]

Basic 1 auto instrumentation has gotten so good in recent years. If you're using OpenTelemetry and everyone should be using OpenTelemetry, all of the common patterns, like all of the models are trained on them. So it is literally faster and easier to build with instrumentation than not to

[Corey Quinn]

and with instrumentation do just once, once I have the code in a compile step or an extra step, it just adds it to the right lines.

[Charity Majors]

This is what's important, right? It's part of just developer intent, right? This is how you declare your intent and that's how you check up on your intent in production. It's honestly gotten so much simpler. You know, I don't I don't fault developers or anyone else for not kind of closing that loop with, with DevOps because the fact is it was, it was prohibitively hard and time consuming and difficult because, you know, you're old school software engineer and you're, you sit down, write some code, you're like, ah, here, I should instrument it and look at it in production. So you're like, okay, I've got a bit of data and I want to do something with it. All right, is it a metric log trace? An exception, an error, a profiling? You know, just. Yeah, okay, if it's a metric, is it a, is it a counter? Is it a gauge? Is it a, you know, just like all down, it takes so. And then, well, what type of data is it? Is it going to have high cardinality? Is it going to be a cardinality

[Corey Quinn]

to worry about and you can blow it?

[Charity Majors]

Yeah, it's just like if it's a log line, which log level do I do? Do I append it to another? Like, it's just you could double, triple, quadruple the amount of time that you spent writing the code, trying to instrument it and then you still wouldn't be done. Like, you deploy it and then it's like, okay, I know the name of the thing that I added, but how do I find it, how do I display it, how do I create a dashboard? It's just like, that was prohibitively, that was really hard. But now we can bring all of this to you, right? In your development environment. It is easier and faster to instrument with telemetry than without it. And you don't have to leave your development environment to go and get it. You know, you could have the agent, like we've built some really cool shit at Honeycomb where it'll just, it'll be like, oh, hey, that thing that you wrote, you know, maybe you want to look at this and you can, you can control how boost it is. You can, you know, but it's, it's right there and that's how it should be. It should be part of your development loop.

[Corey Quinn]

Can we talk about what spans are? Because I'll quote Eric Riddock, who recently wrote on LinkedIn, the basic idea of observability for applications is don't use logs or metrics, just put it all in spans. What are spans?

[Charity Majors]

Spans are bits of, of a, of a trace. I mean, a trace is just structured log with some fancy fields, right? And so the span is the subset of the trace that makes up the entire duration. And I don't know if you've followed any of this, but like the default building block has been the transaction for as long as the web has been around.

[Corey Quinn]

Yeah, that doesn't work anymore with specifically with AI.

[Charity Majors]

Yeah, we just, we just ship something called Timeline that is like, it sits on top of spans. So you know, if, if you, you know, if you, if you run something like intercom, you have got a chat thing and a customer's like, I'm conversation going on. Yeah, customer's like, I'm complaining. You're like, okay. So you spin up an agent, supervisor, agent that spins up more agents. And each of them calls APIs, each of them calls like storage backends and stuff. Then they return and then the customer has another that could span hours. Right. And you need to be able to zoom out and visualize the whole thing. It's super cool.

[Corey Quinn]

And so this is a new primitive that you came up for these use cases where there's a conversation or like a, an LM is involved and you

[Charity Majors]

have like, it's like a meta trace.

[Corey Quinn]

Oh, okay. Yeah, So I guess it's trace of traces. So we need these new building blocks to actually just be able to work with. Yeah, interesting. So I guess this is something to keep in mind like any, any, any engineer who's like building on top of, of LLMs, who is an AI engineer. Now as, as we know, it's either

[Charity Majors]

that or you've just got all these tabs open with traces and you're just copy pasting IDs from one to the next.

[Corey Quinn]

Yeah. Or if, if you're a large enough company, you might have built, or you might have built your own tool, but we know that it's doable. But it's painful.

[Charity Majors]

It's doable. It's painful. I'm really looking forward to seeing over the next few months or year or whatever, just the marriage of tests and telemetry and evals. From a telemetry perspective, with agents and

[Corey Quinn]

AI agents being around a lot of them are now very useful to connect to observability stores. You can go and do stuff. However, one question that comes up is, well, agents have a finite context window and with observability you can really easily overload that. What are approaches you've seen of agents either using honeycombs or some other data sources to make them productive? Have you seen some patterns?

[Charity Majors]

There's a lot of trash data out there and a lot of traditional telemetry data metrics logs, traces where it's all. It tends to fill up your context window with crap. When the most important part of the data is again the relationships between the data. So if you can. And in fact, one of the AI SRE startups posted this great piece a couple months ago about how they see the agents that they deploy in the wild bypass the observability data most of the time and they go upstream to find richer, intact telemetry data. So that's what I would say. Either you give your agents the. But it's. It's the relationships that matter. Right? Because that's what actually helps the AI make decisions.

[Corey Quinn]

And when it comes to observability, I cannot not mention your book Observability Engineering and you have a second edition. Can you tell me why you felt the need to write it and what's new in it?

[Charity Majors]

Oh man, the whole thing is new. So O'Reilly, any. Anytime a book is considered successful and if the topic is still relevant, they'll ask if you want to write a second edition. So it's not really. But I was really excited to write it, the first book. I don't want to say I wasn't proud of it, you know, like, you're not like your children and your books are not supposed to like say anything bad about them, you know, because, uh, it's fine. They're, you know. But it was written 2019-2021. The definition of observability meant one thing when we started and another by the time we ended and there was at no point where I was like, oh, this book is great, let's ship it. It was just like, oh God, I can't do this anymore. Just like, please take it. And I hope that's enough. Now it feels like the definition of observability is more settled. It's everything else in the world that's like changing and crazy. And also I think it's a good book. I hope it can help a bunch of folks. It's got six parts. So the first part is. And I wrote parts one and six. First part is just kind of like grappling with what does it mean to run deterministic and non deterministic systems? You know. And then, you know, my co authors Liz and Austin and George, the part two and three is how. How do you instrument your code and how do you understand it? And there are parallel tracks for doing this with or without AI. And a couple of great guest columns from Jeremy. And then parts four and five are. We have a whole lineup of guest authors and use cases and deep dives. Hanson Ho did one on Front end and interesting.

[Corey Quinn]

Love it.

[Charity Majors]

In mobile we've got some great ones on cicd. Clickhouse did one on columnar storage. Some really, really stellar things. There's a chapter from Kesha and Finn on how they use it iteratively to like do observability.

[Corey Quinn]

Oh, so. So this is, this is a brand new book. It's not. A lot of second editions are like, oh, we added like, you know, two chapters.

[Charity Majors]

This is an entire rewrite and it's twice as long. The first one was 250 pages. This one is 600 pages.

[Corey Quinn]

Okay, so I'm interested.

[Charity Majors]

I'm going to get this book and the part six. It's my baby. And it was originally supposed to be three chapters for observability engineering teams and it turned into. It's a third of the book. It's 200 pages but it's. It's topics for observability governance for, for leaders. And it starts with an open letter to ctos telling them why all their big AI goals are blocked behind their ability to make sense of their system. You know, and then we talk about, you know, software delivery for. No, no buzzwords. Any. Just systems theory. Right? Just if you like Donella, Donella Meadows stuff, then you will like it. And then, and then stuff. And then there's a chapter on how to quantify the impact of observability for your finance. How to. How. How to treat observability as an investment versus a cost center and when you should use observability as a cost center and when you should treat it like an investment because it inherits the type of software that you're observing. You know, and there's a great guest chapter from Rick Clark on staff plus principal distinguished engineers who are trying to drive massive change without authority. How do you do that? And how is observability vital to that? And then there's a chapter on build versus Buy versus Open source.

[Corey Quinn]

It sounds to me that anyone who is inside or wants to be inside a platform engineering team, may you be an engineer or a leader. You probably want to. To read this book.

[Charity Majors]

And at the end there's a chapter that is possibly one of my favorites. That which is. It's called the Art and Science of Vendor Partnerships. And it's just talking about how we can't build all the software that we need. And great vendor partnerships are ones where you have influence over their roadmap and they trust you to do these things and like talking about how most transformations fail. The ones that succeed succeed because someone on the inside has trust and credibility. People believe when you say something, it is true. You know, it cuts through bureaucracy like a hot knife through butter. When it comes to partnering with, you know, the sales org of another company, you do not have trust. Incredibly, you. You work to build trust through reciprocity. You learned just how much you can trust them over time.

[Corey Quinn]

Right?

[Charity Majors]

Uh, but the best vendor relationships are the ones where you genuinely, you feel like their successes are your successes, your successes are their successes. You're happy to see each other because each of you are delighted because you know you're getting something from the. It feels like you are two different teams working at the same big company. That is rare, doesn't usually happen. And that's fine. Most vendor relationships are ones where you shake hands, you exchange money and services, and that's fine. But I think in an era of AI, these are durable skills. These are durable skills for very senior engineers who care about impact, senior engineers

[Corey Quinn]

and also engineering leaders and anyone who wants to become an engineering leader. Because I guess, like, I mean, both of us have been in engineering leadership. Like, you've been in much higher positions than I have. But I think it's fair to say that the way for you to get to that CTO role, that head of engineering, that director of engineering, is to do the work for six or eight or six months a year to year and a half. And to do so, you need to know these things. I feel observability engineering. I'd be underselling this book, I'll be honest the title, but I'm also going to get it and I'll probably think of ways to share a bit more. But thank you for writing and thank you all to all your co authors. But speaking of leadership, I'd love to talk about a little bit of engineering leadership because there's a lot of things that are changing. But I loved one of your very recent takes on leadership, and I'm going to quote you. The most effective leaders are kind, caring humans and skilled business operators. The second most effective leaders are terrible humans and skilled business operators. And after that comes everyone else. There are plenty of good, kind humans who are sloppy operators and bad at business because being good at business is very hard. And you said this in relation to what happened at Twitter. X referring to as Elon as a terrible human but a skilled business operator.

[Charity Majors]

Yeah, I. Well, I don't know that I would call him a skilled business operator. But my point was that Twitter had 16 years to figure it out and everyone could see that they were not Figuring it out and whatever else.

[Corey Quinn]

He is figuring out the business.

[Charity Majors]

Specifically figuring out the business. Yeah, Building products, you know, reaching folks. And you could argue that X has gotten better or worse, but you can't argue that he is running it with 20% as many people.

[Corey Quinn]

Yep. And it's working and it's working.

[Charity Majors]

And some of that, you know, 30 engineers on core, on the core product and another 30 and like 60 engineers, there were 1700 before, you know, and you could argue, and I think it would be true that it's some of the work that those engineers did that but like this is the point. If we don't do it ourselves, meaning hold ourselves to a high standard, build with efficiency, constantly be like trying to get better. We don't do it ourselves, someone will come and do it to us.

[Corey Quinn]

And this is what you also said, you closed saying if we want to remain in leadership, if we want to set the culture and the tone and take the ethical senses that we believe in, we first have to win at the business. And I think this is like, especially now that there's so many changes happening and technology changes, there's whirlwinds, business will go up and down. I guess it's a reminder that like you want to keep your eyes on the prize, which is especially if you're a leader.

[Charity Majors]

The 2000 and tens. There was so much money sloshing around in Silicon Valley and time started to get tough and all of these companies canceled their DEI programs and blah blah, blah. Yeah, they never believed in that. They were just trying to buy people off, you know. And that is very telling to me and I have taken a lot of lessons away from that, which is just that it's not enough to be a good person. I believe that people who are kind and care about people can and usually do do better than sociopaths in the same roles, but only if they're good at business.

[Corey Quinn]

Learn the business, stay close to it. You got to with AI, now that coding has become cheap, now that engineers are running agents. How do you see the role of good skilled engineering managers and engineering directors change? What, what has changed?

[Charity Majors]

Well, the first thing that's changed is I think everyone has to gets to

[Corey Quinn]

be hands on specifically to generate some code, to generate some code, shift to production to some extent.

[Charity Majors]

What it feels like to submit a diff to get a PR through. You know, you should, should know what it feels like. It's just easier now than it's ever been to pick it back up to fill in the blanks, you know, and it's always been the case that leaders were better if they had a hand in it. And now it's just, it's just there's no excuse not to. Teams are getting smaller in general. I think this should be a good thing. If we can figure out how to own more surface area, it should be a good thing. I worry that the way it's happening is, is it's being done by CEOs who are like, oh well, this other company is doing it or it's magic or we're going to do layoffs or like it. And I really dislike the anti management tone. So like, no argument that power tends to drift towards managers over time and needs to get pushed back to engineers. No argument. There's a tendency to have too many managers. You know, the bureaucracy kind of like generates a sort of, you know, it's, it's easier to say yes than it is to say no. And so these things happen. So they need to be pushed back from time to time. But I believe that management and middle management is deeply essential and I look forward to seeing how that works out for them. Not having any. But like the role of a manager, middle management in my view is sense making and context giving because like I don't believe in a world where engineers are just given tasks. Here's your GRA Go do the things AI can do that I want people who understand what we're trying to do, understand how we're trying to do it or who are there to help us figure out how we're going to do it. And you can't engage emotionally, creatively, collaboratively without understanding. And that understanding is incredibly difficult to build and it's fragile and it never lasts very long.

[Corey Quinn]

For those of us listening who are middle managers, it's been a tough few years because what they're seeing is there's a push to have fewer of them, a lot of their colleagues, if they're in unlucky places. They were made redundant and many of them have struggled to get similar positions. We're talking director positions, we're talking head of engineering, senior engineering manager. That role is disappearing faster than ever. I think directors might still be there for folks who are in this position and they do like middle management, they do believe they're good at it. What do you think tactics could be to give them a bit more career options?

[Charity Majors]

Tactically, I would say go back to BNIC for a while. Even if you know it's not what you want to do. If you're at all capable, if you're not capable of it, then I would try to work it. You've got to get AI on your resume. You just have to. And this is a huge career risk. If you're working somewhere where you're not getting these skills, that is a massive risk. I would do whatever I could.

[Corey Quinn]

And this is very interesting that you're saying, get. Get AI in your career. Because I remember about a year, year and a half ago, I started to pay attention to like, okay, this is happening. And I remember a year ago, I wrote an article about how to become an AI engineer. And I talk with engineers who just, like, at their workplace, start to do AI, and now they're AI engineers. Next thing I'm hearing right now is the people who have like two to three years of AI engineering experience are so in demand. I'm doing research, research on a job market, and they're like, this is the best job market ever. However, you know, the people who are like, okay, I have none, but I want to get it. They. And let's say they're out of a job, they're struggling because no one's giving them the benefit of a doubt.

[Charity Majors]

It is really hard. And I'm not saying it's right, but it's how it is.

[Corey Quinn]

And I guess the reason. The reason we're ringing this alarm bell is we know this change has not been as fast. So do it now, because later, do it now.

[Charity Majors]

The next time you go out for a job interview, anyone, you're gonna be asked and you're gonna be filtered out if you don't have it. And the delta between those who are just getting started, most who've been doing was here for a little while and it was very easy to get started. Now it's here and it's. But it's opening. The longer it goes, the more. The harder it will be to catch up. You just gotta get. You just gotta get some.

[Corey Quinn]

Let's talk about directors.

[Charity Majors]

Yeah, directors are usually the ones who. They have been in management for like 10 years usually. And there's a real feeling of fear, often of like, God, tech has changed a lot in 10 years. And. And this is where I would say your body, like, the way we experience anxiety and the way we experience excitement is physiologically almost the same. Like, I used to play piano, right? And before a performance, I'd be like, I'm excited. I'm so excited to do this, you know? Cause I'm like trembling and sweat. But like, the difference is agency. If you sit back and wait for the water to come to you, you're just gonna be freaking out. But if you run towards the waves, if you're like, just like, run towards. Try it. You know, if you have a job now and you're a director and you're afraid of it, it's always seen as kind of noble when managers want to go back to being ICs. I think it's very well respected. Own it, run towards the waves, Own it. Be part of the wave. The frontier of people who are like, I'm so exc. Tell yourself, doesn't have to be true. I'm so excited to be an IC again. It's never been easier to go back and try. I'm going to do it and I'm going to talk about my experience and tell everyone else about just, you got to own it, don't wait.

[Corey Quinn]

And then let's talk about junior engineers. Obviously it's a harder time to guess her as a junior, but how do you think about the value that they bring?

[Charity Majors]

The hardest thing about quantifying the value of junior engineers is if we don't know how to quantify the value of any engineer. So it's all vibes, you know, it's so interesting because I feel like we're over here doing all this hand wringing about will juniors be okay? Will they ever learn the basics? But like my friend Boris who has a new observability startup and he talks to these high school college kids all the time. He's like, they are cooking, they are. They don't know what the software development life cycle is, but they are just like after they are doing so much cool shit. I believe that the kids are going to be okay. We just have to hire them. We just have to give them a shot. They're going to come up with a lot of the conclusions and the ways and the hows that are going to be things that we wouldn't have thought of because. But we just have to hire them. We just have to be willing to give them a shot.

[Corey Quinn]

This weekend, sf, I've talked with a bunch of founders, young startups, and they've been telling me the stories of this open source contributor who was outstanding. So they wanted to hire him or her. Turns out it was a 17 year old kid. They still hire it and now they're telling me like, oh my gosh, the things they do. So I think when you're saying the kids, kids are gonna be fine, just give them a chat and give them a shot. Even if it's an internship.

[Charity Majors]

Yes.

[Corey Quinn]

I feel more company because internship is low risk, low duration.

[Charity Majors]

Yeah.

[Corey Quinn]

And, and even if that person doesn't work out with an interest under their belt.

[Charity Majors]

Yeah.

[Corey Quinn]

So much better.

[Charity Majors]

Totally.

[Corey Quinn]

One question that came up when, when I asked that you're going to be in the show, what I should ask, they said AI fatigue. Like someone, someone asked like, can you please ask Charity as an engineer if I'm starting to get just really, really drained of this? Have you had this? Do you see people having it? And what is a good way to, you know, just deal with it? We know it's here, we know it's here to stay, but it's still, I

[Charity Majors]

mean, my follow up question would be like, which variety of AI fat?

[Corey Quinn]

Okay, tell us about varieties.

[Charity Majors]

You know, because, because some, for some people, when they say AI, AI fatigue, they're talking about receiving slop. Some people are talking about all the hype and the, the, have you heard the, the phrase or the term doom trolling?

[Corey Quinn]

No.

[Charity Majors]

Cal Newport is, I think his name, he's a computer science. He's an AI researcher professor on the east coast and it's his term for what the CEO of Anthropic and OpenAI keep doing about, oh my God, this might be the end of blah, blah, blah. And he's like, it's just doom trolling and they shouldn't, they need to stop it because they're stressing everyone the fuck out.

[Host]

Yeah.

[Charity Majors]

And stop. Because it's just not responsible, you know. So like. Yeah, I think there's a lot of fatigue around that. I think that a lot of people, their family members are afraid. You know, it's just, it's always before in the history of technology, it's been something cool or fun or this will be the iPhone. It'll make your life better. And now it's just like fear, it's pretty crappy. So there's that, there's, there's the fatigue of like I found myself being off social media because I'm just so tired of all of the AI slop posts. It's just like I'm not interested. There are a lot of different varieties here and yes, we are all feeling it. So I guess I would repeat my call for us to remember that we are in control, we are in charge. I think the universal nature of the frustration means that this is a great time to propose experiments where we take back control. Maybe you and your team agree we don't actually want any more AI generated PR descriptions. We don't, none of us use AI on Wednesdays. Maybe we take a week, you know, Just like take control back, try something, propose something.

[Corey Quinn]

I guess because change is so big, experimenting is, has never been easier. And I guess most businesses, most directors, most leaders would welcome teams saying, you know, we're going to try out because their answer will probably be, I mean you're in this position, your answer I guess will be sure.

[Charity Majors]

Better yet, don't even tell me, come and tell me what worked afterwards and

[Corey Quinn]

what didn't and what you learned and

[Charity Majors]

then other teams can learn from that. Right. I think sometimes people are waiting for top down permission but like we don't know what permission to give until it works so much better when it bottoms up, when people are just take control of your time and your calendar and

[Corey Quinn]

I guess maybe we just forgot that there have been major changes in the industry. I remember the iPhone change and I remember the people when the iPhone came out, iPhone and Android, so smartphones, the people who were the Most kick ass iOS engineers, you know who they were, they were typically like 18 or 19 year old kids who went into this and they tried it out. Guess what? Two years later they were the domain experts. The staff Engineer was a 22 year old and then the entry level engineer was a 40 year old. And again, not always. But my point is in the, when there's such big change, you can actually become an expert by very little time by you taking, just taking charge, taking, taking charge. And also no one's really going to tell you no because no one knows what's working exactly.

[Charity Majors]

There's some liberty there.

[Corey Quinn]

So as closing, just to go back to a little bit of being human and slowing down, what are one or two books that gave you something?

[Charity Majors]

Ooh, I really got a lot out of Catastrophe Ethics. I haven't seen it mentioned in many places and I think it might be, I think real philosophy nerds would be like, that's kind of a pop book, you know. And I think the people who are not real philosophy books are like that's kind of a lot of philosophy. But you know, he's, he's a bioethicist, I think. Travis Reader R I E D E R Catastrophe Ethics. And he talks about how the puzzle of modern life is that it feels like everything we're implicated, every choice we make. Are you going to use milk? Well, you know, the cows are tortured. Are you going to use almond milk? Well, water is a problem. Well, you swim like well hormones and it's just like there is no. Whatever you do, you are hurting someone and it feels like the problems are so large that none of our Decisions really matter. And that tension like what? And then he kind of walks through traditional ethical frameworks like utilitarianism and stuff, and just shows how there is no recipe anyone can follow that doesn't lead you to some really stupid.

[Corey Quinn]

And.

[Charity Majors]

And he's like, this is just no gods, no masters, we are. Which doesn't mean that everything's relative. Doesn't mean. What it means is that the way to live an ethical life of integrity is you need to educate yourself about the world. You know, you need, you need to be. You need to know things, right? And then listen inside. You know, where are you drawn? What suffering really speaks to you or what caused you, you know, because. Because no one can tell you what matters. You have to decide what matters. And so that, that introspection and it's so at odds with the sort of performative rage, you know, and that. Which. I'm just so exhausted. All right, so that's one. Number two, this is a book that I recommended a couple times, but I'm just going to keep recommending it because it's so good. It's by Adam Becker and it's called More Everything Forever. And he is a journalist based in San Francisco. He has a philosophy undergrad and a PhD in astrophysics. And he just demolishes all of the AI religion, the singularity and the effect of altruism and accelerationism and the whole like, what if we could have infinite growth foreverism? And he's like, the heat death of the inner of the universe. You guys, literally the only thing we know about exponential growth is that it must end. It must end in an S curve or in a crash. It must end. And he's got this dry sense of humor. And there are a couple times where he's just like describing some of the very real things. It's just like, why do Oxford ethicists want this? He's talking about like taking over star systems and stuff. It's just ridiculous. And he also, he gets in a whack. He's just like talks about all these people who are working so hard on life extension. And he is like, these are a bunch of sad little boys who miss their daddy. And I was just like, oh my God, it is. The oldest fear of humanity is the fear of death. And you just see it, you can't unsee it. So, yeah, those are my two. They're both so good.

[Corey Quinn]

Charity, thank you so much. This finally made it happen.

[Charity Majors]

Finally. It's a good time.

[Host]

I always really, really enjoy talking with Charity. I hope you also liked it. I appreciated how Charity talks about the trust account. If we are debiting trust from the creation of code because AI wrote it and no human read it, then that trust needs to be refilled somewhere else. Testing evals and guardrails are all ways to add more trust that we lost by using AI. I also appreciated how she talked with empathy about both AI camps. The enthusiasts or AI paled folks are seeing the practical wins, while those operating production systems see the slop. Neither side is wrong, but they should talk to each other more so. So if you see wins with AI, share with the broader team but also talk about it when it creates more work, reduces reliability, or when it degrades quality. And for those of us feeling anxious about all this change, especially directors and managers, I'll leave you with Charity's advice. Anxiety and excitement are psychologically almost the same, but the difference between them is agency. So instead of waiting for change to come to you, take charge however you can and make changes yourself. Do change out the show notes below for related to Pragmatic Engineering deep dives on how AI is changing software engineering and for another discussion with Charity on Observability. And I can very much recommend her book Observability Engineering second Edition. If you enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks and see you in the next one.