← Modern CTO
Modern CTO
Why You Can’t Go Off Vibes in Software Development with Stephen Poletto, Field CTO at Span
00:55:50
Loading summary
Transcript135 lines
- [00:00]
A
Today we're talking to Steven Pauletto, field CTO at span, about the modern landscape of software development and how devs are screwing up by going off of vibes. You're listening to Joel Beasley, modern cto. Why are we here, Steven? Are we the caterpillar turning into the digital butterfly? Is that what's going on here?
- [00:31]
B
I don't know. I don't know. I mean, it depends what your p doom is, right? There's like, you're either on optimist that this stuff is going to alleviate human labor and suffering and we're just going to live in a techno optimistic future, or we're all not going to have jobs and the AI overlords will treat us hopefully like pets and treat us benevolently, but they could also treat us poorly and who knows which version of that reality we're living. Joel, the universe, the timeline that we've been living has been very strange. It's fun though. So I don't know which path we're on.
- [01:05]
A
It's fun, it's interesting. It's not boring, is it?
- [01:09]
B
No, definitely not boring. Yeah.
- [01:12]
A
And so what are you doing? You're over at span, what do you do over there?
- [01:15]
B
Instrument development processes. Like, what's the workflow of your engineering team? How are they bringing products to life? And we're at the epicenter of this because AI is changing software development practices very quickly and we're trying to help engineering leaders make sense of it. And so I kind of have a front row seat to this whole transformation, which is, which is, yeah, it's not boring, it's very interesting.
- [01:38]
A
And you're out there and you're giving talks. You have a popular talk called Don't Go Off Vibes. What's that all about?
- [01:45]
B
Yeah, so it's hard to consume anything in the mainstream media about AI and AI's impact on software engineering because you either get the hyper optimistic, it's making us 10x more productive, a hundredx more productive, or you get the doomerism that's like, it's a toy, it's causing all these quality issues. AI slop. And I realized there was no like grounded neutral voice in the market that was just like, let's use data to inform what's actually happening and what's working and what's not working. And so I gave that talk and I'm hoping to continue to study this space, but just be a reasonable non polarized voice, like, I don't have anything to gain.
- [02:35]
A
Get out of here, Steven. No, no, no, no, no.
- [02:37]
B
That's not going to be anything. I know it's not going to do anything for the virality of your podcast. I'm sorry, I'm just going to be a neutral, objective observer as much as I can.
- [02:47]
A
We don't want rational, logical people. No, but it's, it is kind of interesting though because I'm incentivized and we go to write that title to do the craziest, hardest title we could possibly think of that's like, would still deliver like clickbait title, but it still delivers. And we're just if every. And I realized the other day, like, if everyone's incentivized for that, everyone from me to the billion dollar media company, we're all incentivized to be as hype as we could possibly be. What does that do long term to humanity?
- [03:18]
B
Yeah, the erosion.
- [03:20]
A
I don't think it's good.
- [03:21]
B
I mean, you're talking to someone like, I pay for my journalism. Like I get a physical copy of the Economist once a week and I try to read the news. I think they do a pretty good job of outlining the facts of what's happening in the world because of that polluted social media environment and just how difficult it is to like find rational voices in the world. So I agree, I agree. I don't know that it's good, but it's, it's what the algorithms reward. And we're all, you know, having our attention monopolized by these, these systems.
- [03:56]
A
You know, I found, I did an interview with the guy that did a lot of projects with Google X, you know, like the moonshot type projects and all of that. And if you ever go back through and read the history of each one of those projects, it is so cool because it takes these very edgy pieces of technology and then it boils it down, it says, we tried it and here's the results. And it was cost effective or it wasn't and it's going to move forward or it's not. And when I, when I was reading that, preparing for that interview, I thought, wow, it's been so long since I've had just like unbiased brilliant. Like this is just what's happening. Content versus trying to get me to think a certain way.
- [04:34]
B
Totally felt good.
- [04:36]
A
Yep.
- [04:38]
B
Well, I'm trying to be that on the subject of AI transformation and using data and not going off hype and not letting our emotions and polarized headlines drive important decisions about how you staff engineering projects and lead the team through this transformation. But instead instrument use hard Data, take best practices from organizations who are doing it well and just try to make it more grounded.
- [05:05]
A
Who is doing it well?
- [05:08]
B
Ramp is doing it well. They're a very innovative engineering org and they've talked publicly about the agents that they're building and some of the innovative technical work that they're doing. So yeah, Ramp's doing it well. For sure.
- [05:25]
A
Ramp's doing it well. You know, this is good timing for this conversation too because you know, you mentioned one side is 10x to 100x productivity improvement, the other side is doomerism. I always like looking in the market, like what's happening in the market. I was getting early reports that companies were laying, this was like a year ago, two years ago, they were laying off engineering people prematurely. But then you had the guy from Block, I think in Dorsey. Right. He laid off half of his company.
- [05:57]
B
Correct? Yeah.
- [05:59]
A
And he seem, he seems, if you've heard him in interviews, he seems pretty level headed. So for him to do that was actually a surprise to me. He must really have strong feelings about it.
- [06:12]
B
Yeah, I mean there's a lot of speculation about that Block layoff and how much it's a reaction to over hiring that happened during the growth era and a correction for that over hiring and overstaffing. But the messaging that he provided publicly and the public markets responded well to was all about AI efficiency and how they feel like they can do a similar or greater workload as an organization with fewer people. So it is on the extreme end of what we've seen. We've seen smaller layoffs, 10, 20% layoffs from other organizations claiming the benefits of AI productivity. But this was a really extreme one. And it's kind of reminiscent of what Elon did at Twitter when he bought Twitter. And that kind of set off a cascade of subsequent layoffs and market changes and people following the path that he laid out. And I could totally see business leaders looking at what Jack has done and saying, hey, could we do the same thing in our organization? Like how do we derive this level of productivity gain out of these tools and get the same benefit to our gross margins or whatever. The thing that's really interesting about Block is their business is doing well. You know, it's not like they're distressed and doing this layoff in response to business distress. They're doing it in this more offensive maneuver which greatly polarizing. And there's a lot of discussion and speculation out there about it. But yeah, huge news that came last week or two weeks ago.
- [07:47]
A
I can see how it could actually be costing them money by keeping them. And I don't mean just like the direct salary. You and I have been building software long enough to know that there's different types of people that you need involved in the process at different points. And so I've never worked at a massive organization. You have, but I have not. I've always built an idea prototype, turned it into a small team, grew it to about 30 people and sold it off. That's my trajectory. So I've never been in a giant fortune, you know, company. But what I could see happening when these, these teams get really big, and what I saw happening early days with my team is like, there's one type of person who I can give an outcome to. I'd be like, I need you to go achieve this outcome. And then depending on the complexity of that outcome, that spins up several different teams underneath them and all that. But it's almost like it's compressing. It's like now I now, rather than I can get more done if I have fewer people and, you know, in this loop of this outcome we're trying to achieve, and then there needs to be fewer people between them because there's more efficiency in actually getting this hard work done. I could see how it could be a benefit to be smaller and leaner to do something more complicated.
- [08:59]
B
Yeah, yeah. I mean, there was an interview with Brett Taylor earlier this week that, you know, he was describing the atomic unit of productivity as a process. Because almost any business process, or if you think about product development and software development and you think about the stages involved in bringing a product to life, it's like you do market research, you do customer discovery with the insights and the pain hypotheses that you're developing based on those conversations. You turn that into problem statement, problem identification, and then with those problems identified, you go ideate solutions and you test those solutions, maybe in early design stage or prototype stage with the market. And then as you validate the solution, you turn that into a technical specification, you turn that into engineering implementation, and then you deploy rollout and scale. That's a process by which software comes to life. And what AI has the potential to do is compress the cycle time. It has the potential to say, this process used to have this many handoffs and it used to take this many hours or this many days or weeks or months because it has all of these handoffs involved. And there's this coordination cost associated with passing context from one stage to the next. As humans have to get up to speed and understand what happened in the stage prior. And AI has the potential to compress that process. And I think that's a good way to view the world because, yes, large teams, the challenge of large teams is coordination overhead, getting everybody to row in the same direction, getting everybody to have shared context on the goals. And the potential of AI is that you're able to achieve and optimize those processes with fewer people. Now, I think there's kind of two schools of thought at the moment, one being what we saw Jack do with Block, which is, well, hey, we can achieve the same amount with way fewer people. Let's improve our efficiency, our gross margin. Let's just optimize and reduce coordination cost and slim the team. And then there's other people who are thinking, well, hey, if we can do more and we can compress these processes, what more can we do? We've always had a backlog that we've never been able to service because we've always been resource constrained. We've always had these ideas for things that we've wanted to do that we haven't been able to staff. How do we basically promote top line revenue and do more and pursue more of those things that previously might not have cleared the hurdle threshold. Right. Maybe before it was too expensive to build something relative to the benefit to the business. So you'd say, oh, we're just going to discard that or it's going to sit in the backlog, it's not important enough to do. And now if the cost is reducing, maybe the ROI equation changes and it's actually better to go do that thing and create more net new value. So I think you're kind of seeing multiple schools of thought emerge. And what is the future shape of teams? Do engineers feel more empowered to make product decisions? Do product managers feel empowered to write code? How? You know, the two pizza team that we have basically treated as the industry standard for the past 20 years, is that the new standard when people can do more and minimize these handoffs and maybe team sizes can compress? So there's a lot of speculation right now. I have not yet seen kind of like a golden path emerge where it's like, oh, this is the clear answer. People have experimented with multiple different staffing models. Here's the pros and cons and here's the winner. It's very much like experimentation mode right now. And people are. You see people taking different approaches in the market.
- [12:30]
A
There will likely be different approaches by industry and company maturity too, right?
- [12:35]
B
Totally. I mean, it's not too dissimilar from like the COVID disruption, right? You saw people embrace remote. You saw people say, no, we're in office three days a week, whatever. There's this huge debate about the right way to do it. And people chose different strategies for their business and some of them worked, some of them had new trade offs. And like, I think the same thing is going to happen here. People are going to reimagine how should a software development team be staffed in the era of AI? They're going to try different things. And you know, that's the, the beauty of free market capitalism is like, we'll see a competition of ideas emerge and we'll see which ones work and we'll see which ones don't.
- [13:11]
A
I think that's a mistake a lot of people make where they just, they'll look externally, they'll listen to some great interviews on like the modern CTO podcast or something like, and then they'll say, hey, I want, they do that. You know, Steven does that or so and so does that. That's what you know, we're going to do. I'd say, you know, I tend to be very introspective. I always look like, what are the needs of this team in this organization at its current point? Because so many companies are so different. And you can get one piece of advice, but it's from somebody who's 10 levels ahead of you and it means something entirely different to you now than it does to them where they're at. And so you really have to focus on like, what's my team need to right now, what's the best move for my company. And then I just look outside to see if there's anybody that's in my situation that's also experiencing this that has come to the same. I'm trying to validate my conclusion. Like I came to this conclusion for my team. Is there validation here or my way off? And usually through that act of trying to validate my decision that I'm making internally, I then learn some stuff and it shades the decision a little bit. That's how I do it.
- [14:15]
B
That's why every business has different pressures on it, different regulatory hurdles, different product complexities, different markets. And so as a byproduct, you need different skills to navigate those complexities. Some businesses, if you're building a greenfield pure software play product, you might be able to execute with engineering, doing a lot of the quote unquote product work. But if you're in like a highly regulated environment with a lot of compliance and things like that, like, you know, maybe it's better to have a product manager who is an expert in navigating those things or, or what have you. So yeah, I agree entirely.
- [14:59]
A
Yeah. Like if I. When you're talking about the two paths, right, Like Jack at Blanc, he took the. Took one path. You know, I would say from my experience, from the types of businesses that I've run, which are more aligned with Jack's, I would tend to reduce down and become profitable and then intentionally start new experiments and then boot up teams. But you might be in a situation where you're a healthcare company building physical devices and there's very few people that even know how to do this type of very specific work. And at that point you, you might want a strategy where you say, hey, we're going to reallocate this chunk of people to this, these new and we're going to consider this an investment. Like over here, we're going to optimize our core product and get really lean over here. But we're going to take these people instead of letting them go, we're just going to reallocate them for an investment of an area that we know is going to be important because that talent is so hard to acquire and bring up to speed on this. So you could have different strategies.
- [16:02]
B
Totally.
- [16:02]
A
Yeah. But there's not just one answer for everyone, Steven. That would be so easy. Wouldn't.
- [16:09]
B
Would be. I mean, I think what we're seeing with AI transformation in general is there is a set of emergent practices that are unproven, but people are talking openly and publishing their learnings and it's kind of a marketplace of ideas right now. And you see this even with things like harness engineering or what I call constraint programming. Last year it was vibe coding. It was like I'm just prompting the generation of these big swaths of code and features and things like that. And now people are reckoning with the strain that that has introduced in their organization. Whether it's like we're producing too much code to code review and keep up. Human code review can't pace well enough to hold pace with the volume of code generation that we're doing or it's actual quality issues in production. Like earlier this week we saw reports that AWS had an engineering all hands where they were talking about incidents in production that were due to AI generated code and quality issues escaping their quality guardrails and making their way into prod, including like a six hour outage. And so people are dealing with the hangover of just like kind of vibes based programming and realizing, oh, we need to put constraints in place, we need to invest in the environment that these agents are operating in to make sure that they can't, you know, we don't fuck on ourselves and run into these issues. But that's an emerging practice. Like what is harness engineering? What is constraint based programming? It's like, that's basically like a new field that has developed in the last 60 days. You know, you're gonna have to explain
- [17:52]
A
harness engineering to me because I haven't heard this before.
- [17:55]
B
Well, yeah, so harness engineering is the idea that you are delegating workloads to agents. So you are describing product specifications that you'd like claude code to go build, or you're providing a bug report as an input into an agent and having it go try to fix it on its own. And without the right oversight mechanisms to constrain the solution space, who knows what the agent might do. They might go look at your code base, see that there's some pattern of solving the issue somewhere else in the code base and replicate it. But that might be like a tech debt pattern that you actually don't want to be replicated. So instead you want to define constraints in the environment that make it so that when the agent tries stuff, they're more likely to succeed in accordance with your standards. Right. And so this can range from, I mean agents markdown files were kind of the first attempt at this where it was like, well, let's just tell the agent these are our architectural standards, these are our design principles, like please code in this way. But unfortunately, I don't know, you've probably used LLMs a bunch. They don't always follow your instructions. Right. They're kind of like children or junior engineers or something where they go and sometimes follow your instructions and sometimes they
- [19:15]
A
do what they want to do.
- [19:16]
B
They do what they want to do, right?
- [19:17]
A
So they're people too. Steven, don't be racist.
- [19:22]
B
I mean I do think a reasonable mental model of these agents is like they're junior engineers, right? So you need to provide oversight and constraints so that they're going to be successful. And so, you know, if it started with kind of agents markdown files, it's progressed into, you know, how do you do custom linters with remediation messages, how do you do pre commit hooks, how do you have browser automation tests per session so that the agent can validate the correctness of what it's building more real time. So before the agent ever opens a pull request, you have a priori constrained the Solution space such that it can't pass until it does certain things, until it satisfies certain criteria. Right. And so I think there's a spectrum of constraints ranging from things that you express in English language and kind of hope for to things that you make impossible by design. Like you don't give the agent access to RAW SQL, it can only go through an interface layer where you control the way that it can mutate the database, for example. And so all of these kind of practices of how do you create an environment of constraints and then under those constraints the agents can thrive. That whole practice is kind of being referred to as harness engineering.
- [20:38]
A
Interesting. Have you come across Promise Theory yet? It's, it was new for me.
- [20:45]
B
No, no, no, I haven't heard that one.
- [20:46]
A
Yeah, yeah, I, I would butcher it. I had the scientist PhD people on that that came up with it and then a company that had used the theory to implement it in swarms of agents. But the TLDR is they've experimented with different ways to get the agents to do what you want them to do without derailing. And the way that they did it was fascinating because they, they tried a bunch of ways that didn't work control and then like all of these different things like demanding it do certain things. But then it found they found a way to get it to do what they wanted it to do and we did a whole episode on it.
- [21:28]
B
Very cool. I'll have to go listen to that episode.
- [21:29]
A
I hadn't heard that one.
- [21:30]
B
Because there's so many emerging practices, right. It's like spec driven development agents markdown and how to get the most reward for that, this constraint based programming or Harness engineering. I'm seeing new frameworks like bmad, which is like how you apply AI to every stage of the product development process. So you can actually have agents go do stakeholder interviews and like user interviews to help frame and shape the product requirements that you're defining. So there's. Yeah, it's a marketplace of all these different techniques and approaches that people are trying and it's, it's a fun time to, you know, very intellectually stimulating. So yeah, I haven't heard of this Promise Theory, but I'm going to go check that out.
- [22:16]
A
I think the endpoint is fairly obvious. I think how we get there is going to be exciting and we're not going to be entirely certain until it actually happens. But I do think the endpoint is basically we just have these interfaces where we tell it what we want and it gives us what we want. I wanted to do this, and then it just comes back perfectly and does it right the first time. I think that's where we're headed. I don't know if it's going to be 5 years, 10 years, 100 years.
- [22:43]
B
I think. I mean, I think we will get closer and closer to that. Something that I believe is that taste and judgment has not yet been expressed by these systems. It's entirely possible that it will be. And then we really are just expressing high level intent and offloading everything. And that's a very different world than the world we live in today. But at least today I feel like there's still a level of judgment and taste about, like, which problems are worth solving and what does a good solution look like and how do you shape that into a cohesive product experience for your customer? The CEO of LINEAR was tweeting about that earlier this week. He was saying, you know, the cost of shipping features is now so low that they're shipping too many features and the features aren't cohesive and the product experience is degrading. And it's like there's like this taste and judgment layer that sits on top of everything that curates into something good. And I don't know it's entirely possible that we'll get to the point that agents can do that on their own, but I think for the time being,
- [23:52]
A
you think we will 100% because it already happens in life. So imagine you're a king of a very wealthy nation. You just express high level intent to. One of your head officers would be like, I want this to occur. Now. How many thousands of people are part of that pyramid to make the outcome occur? Who knows? But.
- [24:13]
B
And if you don't like the outcome, you fire the person, right? Or depending on the era, behead them or what have you, or they come
- [24:19]
A
back with an outcome and then you provide more input to get a better outcome. And so that that already happens. The thing is, historically, information itself and that ability to direct large amounts of intelligence, whether it's human or silicon based, was for like the most wealthy people, right? Even way back in history. Just access to books itself or access to experts. We can turn on podcasts now. And so you couldn't do that 100 years ago. You'd have to know these people and bring them in and talk with them. And so what we're doing is we're giving that to the average person. And I think it's a brilliant thing. I'm not like 1000% optimistic about every little thing that'll happen from it. But I think there's a ton. It's like a knife. It can kill you, it can feed you. Like, it can do a lot of different things.
- [25:11]
B
But in that world order, what do you think we spend our time doing as humans?
- [25:18]
A
So. Good question. So humans, they. I like to think about money a lot because money is a way that humans exchange value with each other. Like, we don't give dollar bills to like platypuses or like, or elephants or something. Like they don't have any use for our money, it's just paper to them. But money is how we exchange value with each other and how what we value over time changes. And so I'm certain that whatever we value today, we probably won't value a ton of it identically in a hundred years, but there will still be the transfer of value among humans. So that's what I, when I think about the economy, I don't think it's going to go to Z. I don't think it's crashing to zero. And like there will be no economy and no way to trade value. I think there's going to be incredible amounts of peaks and valleys within different industries at different points in time that we as a civilization are going to have to try to figure out. Like, it's going to be hard. Like imagine, Stephen, you wake up tomorrow and there's this new tool that comes out that completely automates everything that you and your specific niche job does. And then now you're going to have to find a new value thing.
- [26:28]
B
Yeah, yeah.
- [26:29]
A
In the marketplace to do.
- [26:30]
B
And if you think about like the, the animal kingdom, right, and the, you know, superiority of Homo sapiens, the, the reason that humans have become the top of the pyramid is due to our intellect, right. The ability for us to coordinate, communicate well and our intellect to orchestrate, you know, behaviors at scale. Like that is what, you know, made us kind of masters of planet Earth. You know, go read Homo Sapiens. It's like great book. Like I love that book. I think the challenging thing is if we are developing an intelligence that can out compete us, there becomes a very existential question about like the value of human. Like what is the unique value that humans have that is not just intelligence because we've built these systems that can take our niche jobs in the intellectual world. And that's a pretty thorny existential question.
- [27:26]
A
Yeah, it'll be interesting, but the humans are asking it and then we gotta be concerned when the robots start. You know, my concern levels vary for some reason. I have maybe it's just because I am a human. But I have more of a concern when the optimus bots start rolling in the streets than I do right now with software and stuff like that. Because as the AIs take more physical form, I think our risk factor goes up.
- [27:52]
B
Like, totally. Yeah, yeah, yeah. There's still a lot of physical world stuff that we can do that machines can't do. Even if they can take vast swathes of intellectual labor, there's still like a lot of human services that are in the physical world. Totally, yeah.
- [28:05]
A
Like right now, if we needed to blow up all of our data centers, we could find another form of currency and find. It would be rough, but we could figure it out and we would still survive and humans would be fine. We would figure it out. That being said, when we get to a point where there's two optimists to every one human roaming the planet, right at that point, we can't. There's not. You can't just nuke the data centers. You have this decentralized autonomous robot thing that can kill you that outnumbers you. That's the point. To me, that's scary.
- [28:40]
B
Yeah.
- [28:41]
A
Because even if there's a human bad actor controlling them same. It doesn't matter if it's everyone's so fantasy full or like, what. They always fantasize about this idea that, like, the robots go crazy. I'm not really concerned about the robots hallucinating going crazy. I'm more concerned about the person who's evil, who's directing the swarm of robots going crazy.
- [29:01]
B
Sure, sure, totally. Joel, do you normally get into these existential futuristic topics with your other ctos or is this.
- [29:09]
A
We do okay. I love it. It depends on the day. It depends on if we feel like
- [29:15]
B
as of late, this is like, this is what's in the ether of the world. You know, everyone's grappling with these questions because we're seeing the real impact and the real potential. Like, I think what happened last year was everyone thought these things were toys, at least in the software development world. It was like, oh, I tried it. It kind of works. It's a toy. And then some models came out in the fall, we had like Opus 4.5. I think a lot of people had actual time where they weren't in the Grinder and in the execution of their business over the holidays and were playing with these things and like entering 2026, it feels like a new world order. It feels like everyone is like, wow, how do we become an AI native software development Loop, like what does that mean? How do we change roles and responsibilities? How do our processes evolve? What does it mean to leverage these tools? And all of those questions just feel different than they did. Like middle of last year.
- [30:14]
A
You're exactly right. And I think you nailed it too. When those breaks happen, that's when I get to stop running the business and play more, you know, that's when I pick up these new things. Oh, I've got a, you know, three days. Because of the way the holidays run. I've got three days. I don't have anything I have to do. My team's kind of checked out. Like maybe I'll experiment with this. And over two years that happens here and there a couple times. And everyone who's currently certainly in executive leadership or even, you know, just any position in the company, if in the past two years you've been able to see the progress of the models in a way that like everybody has a different industry. So we're always watching our industries and like watching them progress. It's very slow, you know, if you change happening, it's very slow. It takes a long time to watch it over 10 years is like you've been in the game to watch it over 10 years, you know. But we saw a massive shift happen and it gets significantly better and better. And in like a two year time span from the first time GPT kind of came out to like right now, it's been like three years. I think two or three years. We've seen it get so from going hallucinating and off the rails and stuff in the early versions to now, to not to not being connected to search engines to now having real time search engine. Like we, we've been watching these things progress and get so much better. And we're, we're all thinking the same thing. We're like, oh wow, this is, this is not gonna slow. This, these trains don't slow down. I think they're gonna keep going.
- [31:44]
B
Yeah. And I mean, I think what we're seeing is a year ago, two years ago, they felt like autocomplete. They felt like a search engine at your fingertips. You've got stack overflow in your ide. You know, like that was kind of the mental model. And now it's like, well, customer reports a bug. We've got a Gentix subsystem that checks out. The bug reads our agents md, you know, goes and drafts a fix. If it passes all of the tests and our linters and gets through the harness, the environment that we've constructed for just opens a pull request. And now there is no human reviewing that bug. It was just like, a customer wrote a bug report. Here's a PR to fix it. Like, that's a. That wasn't happening two years ago. That's. That's a totally different world.
- [32:39]
A
And it's. It's fairly reliable in my. In my experiences with really small things that are very specific that can be validated. Like if I had a queue that came in and I was like, yep, that's valid. And then it proposed to me the situation as a pull request, and I could look at it and say, like, all right, cool. And I just let it go into production. I think where everything gets swampy is when you do, like, larger features or things that impact the system. But on pull requests for bugs and things like that, I think it's pretty sharp.
- [33:09]
B
Yeah, yeah.
- [33:11]
A
What does SPAN do and how can people buy it? Yeah, well, let's make sure we do work here, too. We're having fun. Yeah.
- [33:17]
B
No, span, it integrates to your development tool chain. So we connect to, like, your source control, your ticket system, your genai tools that your team is using, so we can understand code workflow, and then we can understand AI tool engagement and the patterns around using those AI tools. And with all of that context, we're constructing a work graph of, like, what major projects, themes, and initiatives is your engineering team working on and how are they flowing through the development process? So we can help identify bottlenecks, we can help identify investment envelopes, like, roughly how much time allocation is flowing into different projects. And so it's this software engineering intelligence layer that helps leaders understand where is human time and token spend going and where is it encountering friction and bottlenecks? And so this has been increasingly valuable as people are embracing AI and wanting to understand, like, how is AI reshaping the development loop? Where is it creating new challenges? Where do I need to be focusing my attention? So that's what we do. That's what span's all about.
- [34:33]
A
So what is the problem people are facing when they find you? Is it out of control? Token spend? Is it not?
- [34:38]
B
It's a mix. Yeah, it's a mix. You know, a big one is like. Like, we're spending all this money on AI tools. We're having a hard time quantifying the total impact that it's having. Can you help us understand the roi? Can you help us understand how much more productive we are? And are we actually more productive or do we just feel more productive? Where are there New challenges emerging and how do we tackle those challenges? Who is proficiently using these AI tools and who is kind of shallowly using these AI tools? How do we get everybody in the organization to become proficient? So it's those kinds of questions. It's basically organizations who are like, we've seen the light. We think that the companies that are introspective and have an attitude of self improvement about navigating this AI transformation are going to outperform the ones who just kind of go with the flow. And so USPAN can be this lens of data and consulting and services that help us figure out how to optimize our AI development.
- [35:46]
A
That's awesome. So you do some tool, you have tooling, but you also have like consultative side of things too.
- [35:51]
B
We do, yeah. We're spending a lot of time talking to customers, understanding practices that are working, doing like benchmark reports so we can aggregate and anonymize data across our customer base and use those to understand macro trends so you can see like how you're doing relative to the industry at large. And we use all of that data and anecdote data to help companies move through this transformation process.
- [36:17]
A
Interesting. Now there's this study from Meter. Is that how you say it? Metr? Yeah, tell me about that study.
- [36:24]
B
Yeah, it was published last year and so obviously things are changing quickly. But they ran a experiment on an open source project where they gave access to AI tools to a subset of contributors. Kind of random, you know, basically you would be assigned a task and then you would be told like you can use AI or you can't use AI for that task. And then they tracked productivity and self reported productivity based on that assignment, like basically trying to do a randomly controlled AB test. And what they found was like engineers self reported significant productivity gains when they were given access to AI tools. But when they actually looked at cycle times associated with the completion of the projects and other productivity stats, they saw a degradation in productivity. So there was this paradox of people feel more productive but they're not actually more productive. Um, so yeah, pretty, pretty interesting study. There's, there's some more recent studies that they're working on too.
- [37:37]
A
I would like to see, I would, I like to poke at all the studies. I don't care what you, what the information is. I like to, I always like to rip it apart because I'm a nerd and I would be curious to figure out how they controlled for experience level. Like, because the thing like I can go in there inexperienced, do it, learn something, and then feel like, oh wow. The next time, like I could feel more productive and I could actually be more productive in a future task because I learned like did some of these people, were they learning how to do it for the first time with AI or were they already like integrating it into their. That that's the control. I would totally, that's the metric I'd want to see is how many hours a week are all these participants already using AI versus this is their first interaction with it. Because I was horrible. I was, I, I just, I dismissive of it at first after being a software engineer for almost 20 years. And then I, my, my, one of my best friends, he sent me a YouTube video of some dude like really doing it, like showing you how, like showing another senior engineer how to use it. And so I watched that and I was like, oh, I was, I was using this tool wrong. I didn't understand how to use this tool. And then it, it just made me crazy productive.
- [38:52]
B
Yeah, that, that's the challenge with a lot of these studies is like there's multiple conflating factors so it's hard to control for everything. You know, even in.
- [39:00]
A
It's great for sales.
- [39:02]
B
Yeah, yeah.
- [39:03]
A
Well, I mean I do, I do agree with the sentiment of it though. There's definitely people out there who feel like they're faster and they're slower. I, I bet that is true 100%.
- [39:11]
B
I mean what we're, what we're trying to do at span, because there's a lot of. You could do like a longitudinal, like a time based measure of productivity where you could say look, we, we've rolled out a tool or we gave like this cohort of engineers a tool but not this other cohort. And then we could track the impact over time. But that also introduces a ton of conflating variables because the environment, the work that you're doing three months down the road relative to that point when you started the experiment is very different. What we have been trying to do at SPAN is say estimate the dosage of AI in any given pull request, how heavily AI influenced was a given code change, and then track all of the workflow signals around pull request, lifecycle number of code review, comments, subsequent rework that happens after that pull request is merged. Is there like defect fixing and bug fixing that happens in the 90 day window after the original merge. And so now you have something that is more controlled. It's like how much AI was actually in that given pull request. And you can cohort on a pull request basis. So that's been the foundation for a lot of our benchmarking and productivity insights because we think it does a pretty good job of controlling for all these different variables. And that's where we've been finding some really good insights around, like, actually, code review burden is way higher and there are more rounds of feedback happening in code review as a byproduct of AI adoption. And I think a lot of attention is now being given to code review processes and workflows and how to shift left, you know, quality guardrails so that you're not catching them at the code review stage, but you're defining them in the environment earlier with linters and things like that. But yeah, that's kind of been our methodology. It's fun to see the different study designs and then it's fun to try to figure out like the best methodology that we feel can control for the most number of variables. And this is what we've landed on
- [41:11]
A
and I like that. One of the things that stood out to me about your product as you were talking is the consultation type deal. Because There are many SaaS type tools that are like, all right, we'll digest your repos, we'll do this and then we'll give you a dashboard. And it's like, well, that's great. But I think a lot of the people listening to this show, or at least a lot of the high level technology leaders, they've got a lot of other stuff going on, but they might want to figure out this token spend and human time and like, what's going on is it? And so they would just hire you guys and then you guys would have tell them, you guys would look at the data, help interpret it. Am I understanding that correctly?
- [41:48]
B
Yeah, I mean, you're right. Engineering execs are busy. I mean you, I've been an engineering exec, you've led engineering teams in the past. You know how fragmented your attention gets. You're like managing people, you're doing project execution forums, you're doing exec reporting up to the CEO and the board, maybe, you know, cross functional syncs with sales and product. Like you're busy. And so yeah, the idea is that we get a set of out of the box metrics by using our software, but then the signals ultimately inform recommendations that we can make where we're like, hey, here's how you're doing relative to the industry. You know, it looks like you're struggling with xyz, for instance. One of the things that we are seeing across a number of our customers right now is like Elevated code review burden. So we're like, hey, here's some themes of repeated code review feedback that's happening in your AI pull requests. Here's some ideas for things that you could do to streamline that. So just trying to give targeted recommendations because, yeah, our customers are busy and they just want to know, what should I do? Yeah.
- [42:53]
A
And as harness engineering kind of matures, you can watch it evolve and share knowledge across your customer base.
- [42:59]
B
Totally.
- [43:00]
A
Yeah.
- [43:01]
B
Yeah. It's a nascent practice. I mean, it's a nascent practice in some ways and it isn't in others. If you think about, like, hypergrowth organizations that have needed to scale up and hire lots of humans, you also need to do this practice of environmental constraints so that the junior engineer you just hired doesn't take the site down. Right. Like, people are going to make mistakes. You assume people are going to make mistakes. There is knowledge trapped in the heads of your, like, senior tenured engineers. So what do you do? You write tests, you know, you know, create ways to roll back safely when failures occur. You contain the blast radius. You invest in linters, you invest in code standards. You do all of these things. It's just now every organization just hired a hundred junior engineers. And now this stuff is even more important. And so we give a fancy label to it called harness engineering, which is really just guardrail design and safe engineering environments.
- [43:55]
A
Do you have kids?
- [43:57]
B
I don't have kids.
- [43:58]
A
Oh, man. There's so many parallels between that and having children because, like, how many mistakes you let them make and all of that. I would promise you having kids will make you a better technology leader. I've been a fan of that for a while. For about eight years.
- [44:14]
B
I believe that. Yeah, Yeah, I believe that.
- [44:17]
A
Okay, so you were at Lattice. They grew from 15 million to 125 million. ARR. Your director of engineering at Dropbox for eight years. You've done it all. You were Apple, XCODE stuff. You are like a nerd. How did you run into span? Why did you decide to join them?
- [44:35]
B
Yeah, well, so Jay Zak is the CEO and co founder and I worked with him at Lattice. He was leading product when I was leading engineering and I really liked working with him. Like, he's an incredible human. He's really talented, really smart. I learned and grew as a human working with him. And when he started span, I was fortunate to be one of the early design partners. Like, I used the software when I was still leading the engineering team at Lattice and provided a lot of product input and feedback and kind of helped shape some of the early product into what it is today because I was an actual practitioner, like using it. So yeah, just like good old fashioned human relationships and working with good people. That's how I ended up doing that.
- [45:22]
A
Do you lead with that on sales calls? Because you should.
- [45:25]
B
I do, yeah. I mean I tell people, yeah, they're like, like, I'm a former cto, I'm a former engineering leader. Now I'm doing a customer facing role which is very different. How did I end up here? Like, this is a weird career path for someone to take. Right. And it's like, because I worked with the early team and I found the product space compelling, I found the challenges intellectually stimulating and yeah, I just wanted to be part of it, you know, so. So yeah, my role now is very different. We have a co founder and cto, Henry, who leads the product development efforts. He oversees the engineering team. I provide ample input and I'm an important stakeholder to that process. But I'm not leading the engineering team at span. So I'm leading go to market and doing sales and customer support and the consulting stuff that we were talking about. So it's a totally different role and it's been fun. There's been a lot of learning. There's been a lot of things I don't like about the role. But. But it's been fun. Nice.
- [46:30]
A
Yeah, it's, it keeps it interesting. Right?
- [46:33]
B
Careers are meant to be interesting. I think like you want intellectual stimulation, you want growth, you know, yes, there is a financial component to careers and needing to support and provide for yourself and your family. But, but beyond that, it's also just like I want to grow as a human. So how do I do that? How do I satisfy both of those criteria?
- [46:56]
A
Yeah. How do you wake up and enjoy life? Where do you get your variety from? And at some point you have to have. At least for me historically, I'll do something and I can stick with it for several years, but eventually I'm like, okay, I have approached this from 99% of every possible angle I need to solve a new problem because I found that I'm most energized when I'm doing something difficult.
- [47:22]
B
Yeah. Yeah. And I will say, like, if I, you know, careers are long and who knows what I'll do next after span, but like, if I decide to go back into product development and leading an engineering team again, I will be a significantly better engineering leader. As a byproduct of understanding how sales and marketing and customer success and customer support Works way more deeply because like the empathy level has just gone way, way up for the challenges on the business side of the house. So, like, to me it's just a net good of like. Yeah, I think this will eventually make me a significantly better engineering leader if I decide to go back to that in the future.
- [48:01]
A
You'll probably go coo, CEO. Who knows, who knows, who knows, who knows?
- [48:06]
B
We'll see. We'll see. I don't have, I've never had a long term plan with my career. You're right that I have had a, a good career trajectory to date. I'm very proud of the things that I've done. But none of that was planned. Right. That just, that just happened as a function of me chasing impact and intellectual stimulation.
- [48:25]
A
Ah. And you just said the magic words that I've done almost a thousand of these interviews and like the top 1% of the top 1%, that's what they say. That's what they're doing. And they're trying different parts of the business. They're always doing interesting things. They're trying to stay curious. They are just trying to wrap their mind around it. And a lot of them also say they didn't have like some very specific, like, I'm going to be this role by this time. And this role, they just like, where can I be helpful? Yeah. I'm curious, what's the problem happening? How can I add value right now? Like, what can I do? Like, what's going on, you know, and then just doing it?
- [49:05]
B
Um, yeah, that was my, that was my path. Totally. I did not have a plan. Yeah.
- [49:10]
A
Yeah, I like that your LinkedIn title says CTO ish. That's the, that's my favorite thing about you, by the way.
- [49:17]
B
I mean, how do you describe someone who was a CTO and is now doing sales and customer facing stuff? And how do you. I don't, I don't know. I didn't know how to describe it. I was just like, I'm CTO ish. I'm not actually CTOing right now, but I'm talking to CTOs every day, you know, and mapping what's happening in the software engineering landscape. So I don't know. I'm CTO ish. Yeah.
- [49:43]
A
And who was the CEO that brought you in?
- [49:45]
B
Jay Zak. Jay Zak Stein jzach.
- [49:48]
A
Okay. So one of my favorite things about Jay Zach, I don't know him, but is that he was able, because he worked with you, he was able to see and understand you and how you make Decisions and how you solve problems and know that like he could put you, you in this role and that you would figure it out. Like you would problem solve it. You would figure it out. You would speak up if you were underwater because that's the big thing we all know is like real hard to get people to do. You would, yeah, you would self report if there were issues. And so that says a lot about your character too, that somebody that had worked with you and you were an expert in one, one specific niche field would let you join the company and actually let you be in another part.
- [50:29]
B
Totally. I mean, I don't think in the abstract when I was leaving Lattice that anyone would give me a head of go to market interview.
- [50:38]
A
Right, right.
- [50:39]
B
That no one would look at my resume and be like, ah, he's the guy to hire for this, you know, so, so yeah, I mean that's, that's the beauty of, of deep human relationships and the trust that you build with people. And you know, the thing about the technology industry is like it's smaller than you think it is and, and just trying to do earnest good work and maintain good relationships, they serve you in really unexpected ways down the road. So I've heard ample stories of folks who co found or do early stage startups on the basis of those good historical working relationships. And the upside to that is that a lot of this stuff is relationship based. The downside is it can sometimes be hard to break in as an outsider. Right. But yeah, you're spot on that. I'm very fortunate to have been given the opportunity to do something totally different and I take that seriously. To just try and do earnest good work and do my best and I'm not succeeding at every aspect of the job. I'll tell you that. There's some stuff that's hard, but that's the point of. To your point, you communicate as a team. You try to figure out, okay, if I'm not good at these things, how do we kind of cultivate an organization who can compensate for those things so that I can focus on the things that I'm best at? And it's again that impact orientation of aligning. I mean it's kind of the Japanese concept of ikigai, I'm sure you've seen.
- [52:00]
A
Yes, 100%.
- [52:01]
B
Yeah, it's like trying to align those things of like, this is what I'm good at, this is what I can have impact doing. This is what the world needs from me and just try to figure out what that shape of role can look like. Over time.
- [52:13]
A
Well, and the hard part, the reason why it's an art versus a science is because it changes. True.
- [52:19]
B
Yeah. Your interests change. Totally.
- [52:22]
A
Totally. You can be fully aligned in one season of your life and then find yourself out of alignment in another season.
- [52:28]
B
Like, five years ago, I wanted nothing more than to, like, lead an engineering organ. I was given that opportunity to go do that at Lattice. When I was leaving Lattice, I was like, I don't want to do that right now. I want to do something different, you know, and totally. So your interests, your gravitations change. So it's a constant journey of just trying to align those things. Yeah.
- [52:47]
A
And I think where I got in trouble early in my life is when I ignored that and tried to force other things that I think this is the way it should. So I'm going to force this versus listening to. This is where my attention and energy is flowing.
- [53:02]
B
Totally.
- [53:04]
A
Dude, this is awesome. What's the next step people can take with Span? Do they go to the website? Do they?
- [53:08]
B
Yeah, Span app. Check us out. You'll see an overview of what we do. There's a lot of really exciting stuff on the roadmap, so if you want to follow along on our LinkedIn, we'll continue to be publishing some really cool updates over the next couple of months, and the hope is that we'll be doing a combination of really cool software stuff and product stuff, but also surveys and insights from the market. So, yeah, follow us along and hopefully we can add some value and some color to this emerging conversation around AI's role in the software world.
- [53:41]
A
You nailed it. Hey, tell your design team, send them a DM after this and be like, joel loved the design, the ui, everything looks. It looks so beautiful. It's some of my favorite typography. I really enjoyed it.
- [53:52]
B
I will absolutely do that. And that's Jared Arandu and team. He's our design lead and he does some phenomenal work.
- [54:00]
A
Yeah. Yeah. Good job. It's beautiful websites, beautiful software, so it's worth. Worth checking out. Where are you guys located? Are you in Austin?
- [54:10]
B
We're distributed.
- [54:11]
A
Are you distributed?
- [54:12]
B
Yeah. The company headquarters is in San Francisco. Our founders are in the Bay Area. We have a small office there, but the team is distributed kind of all over the place.
- [54:23]
A
I don't know why I said Austin. For some reason I had that in
- [54:25]
B
my head, but maybe I just gave you Austin vibes. I don't know.
- [54:29]
A
I don't.
- [54:29]
B
Is that what it is?
- [54:30]
A
You've got some Austin vibes. Hey, we don't do Vibes. That's the title of this episode.
- [54:34]
B
Yeah yeah yeah.
- [54:36]
A
Thank you so much for listening. And if you found this episode useful, please share it with a friend or a colleague who you think would get value from it. And if you if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email. Joeloderncto IO Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.