Podcast Summary: The Pragmatic Engineer – "Measuring the Impact of AI on Software Engineering" with Laura Tacho

Host: Gergely Orosz
Guest: Laura Tacho (CTO, DX)
Date: July 23, 2025

Episode Overview

This episode dives into the real-world impact of AI tools on software engineering, cutting through media hype to background data, trusted case studies, and the practical realities of how developers are working with AI today. Gergely Orosz and Laura Tacho explore how to measure the true effect of AI on engineering productivity, developer satisfaction, and business outcomes. They discuss adoption trends, what AI actually accelerates (and what it leaves behind), and how organizations should be measuring value. Especially relevant are insights from large companies like Booking.com and Workhuman and tactical advice for engineering leaders who want to roll out AI successfully without falling for misleading metrics or overblown expectations.

Key Discussion Points and Insights

1. Media Hype vs. Reality

• Sensational Headlines:
  AI in software is frequently overhyped in mainstream media, with headlines suggesting imminent mass-replacement of developers or that AI writes enormous portions of production code.

• Quote:
  “Those headlines suggest that 30% of Microsoft’s code that’s running in production was authored by AI. That is not at all realistic…we’ve never seen data consistent with that kind of sensational claim.” (Laura, 05:29)

• Oversimplification:
  Such narratives often misunderstand how AI is used or measured, confusing code completions or autocomplete assistance with genuine code authorship and impact.

2. Measuring AI’s Impact: What Actually Matters

• Inadequacy of Simple Metrics:
  Metrics like lines of code generated or “acceptance rate” (the rate at which code suggestions are accepted) are misleading and don’t reflect productivity or business value.
• Quote:
  “Source code is a liability. Do we really want to measure AI impact in terms of lines of code generated? I certainly don’t.” (Laura, 16:03)
• Laura & DX’s Measurement Framework:
  • Utilization: How many developers actually use AI tools (measured by DAU/WAU as a % of the population, and license allocation).
  • Impact: Developer experience improvement, time savings (esp. on system toil or debugging), actual velocity increase.
  • Cost: Tooling cost, licensing, and particularly the new challenge of token-based/consumption pricing.
• Quote:
  “Acceptance rate is just such a tiny part of the story…we need to track the impact across the lifecycle, and really stay focused on the end result – more revenue, reduced cognitive load, better developer experience.” (Laura, 14:11)

3. Adoption Patterns and Use Cases

• Case Study—Booking.com:
  • Major focus on adoption and enablement (office hours, training, leadership sponsorship).
  • Achieved 65% weekly/daily use among developers—well above the industry median.
  • Not all devs use AI tools, often due to license constraints or because tools are poorly suited to very novel or specialized code.
  • Quote:
    “It’s not necessarily that these individuals are skeptical, some of it is just that their organization doesn’t make a license available…And for some product areas, it’s just not that effective.” (Laura, 20:37)
• Top Use Cases (Study of 180+ Companies):
  • #1 Time Saver: Stack trace analysis and error debugging
  • #2: Refactoring existing code
  • #3: Mid-loop (inline) code generation
  • Other important uses: code documentation, brainstorming, planning, unit test generation—often more valuable than just generating net new code.

4. The Paradox of Time Savings and Satisfaction

• Developer Satisfaction Risk:
  When AI accelerates the “fun” parts of coding, developers are left with a greater share of meetings and administrative toil, which can reduce job satisfaction.
• Quote:
  “Many developers were actually feeling less satisfied because AI is accelerating the parts that they enjoy. And so what was left over was…toil, meetings, the administrative work.” (Laura, 00:04 & 30:08)
• Coding Is Not the Bottleneck:
  Developers often spend only ~20-25% of their time actually coding, so time saved by AI here doesn’t necessarily translate into dramatic business outcomes or headcount reductions.

5. Architectural and Documentation Shifts

• Codebase Design for AI (and Humans):
  • Emphasis on “clean interfaces” and discoverability to make it easier for both agentic models and human engineers to navigate and use services.
  • “Write documentation for both AI and humans”—lean to examples and easily-ingestable formats, not just narrative docs or screenshots. Companies like Vercel are seen as models here.
• Quote:
  “The Venn diagram for what’s good for AI agents and good for humans is a circle—clear boundaries between services.” (Laura, 40:47)

6. What Leading Companies are Measuring—and Reporting

• Case Study—Workhuman:
  • Used the AI measurement framework (utilization, impact, cost) with pre/post baseline data.
  • Found an 11% boost in org-wide developer experience, and daily/weekly AI users had 15% higher velocity than non-users.
  • Sustained benefit requires both technical measurement and self-reported developer experience metrics.
• Advice: Start measuring now, baseline first—don’t delay even if you don’t have perfect data.

7. Evolving Tooling Costs and Investment Patterns

• Tool Pricing & History:
  Companies once easily spent $3k–8k/year per developer on “core” software tools. We may be entering a similar era, with budgets for AI tools (possibly $1200–$2000/year per dev).
• Token-based Consumption:
  Key new challenge: who gets access to more AI power—junior or senior devs? Where does ROI per dollar spent show up?
• Comparison with Past Hype Cycles:
  This AI wave is similar to the container/Kubernetes era: chaotic at first, then gradually stabilized through industry consolidation.

8. Engineering Best Practices for AI Rollouts

• Highly Regulated Industries Lead the Way:
  • Financial, insurance, and pharma companies are deploying AI more methodically and seeing better results due to required structure and policies.
• Case Study—Indeed:
  • Methodical experimentation with different tools.
  • Segmenting use cases (e.g., using AI reviews to speed up feedback loops across geographies).
  • Treating AI adoption as a controlled experiment rather than a one-size-fits-all deployment.
• Great Use Case—Automated Migrations:
  • Much hated by devs; AI can do heavy lifting and present ready-to-review PRs for upgrades, saving time and reducing monotonous work.
  • Prompt engineering tip: manually perform one migration, then use the diff as an example for more accurate multi-file AI migrations.

9. What Should End Users and Companies Expect?

• Faster Time to Market:
  AI may enable rapid experimentation and shorter time-to-market for validated features. The companies that already excel at experimentation (A/B testing, product iteration) will benefit most.
• Quote:
  “I think roadmaps are on their way out…companies that are going to win with AI will focus on rapid experimentation.” (Laura, 57:11)
• Potential Risks:
  Feature bloat and instability if new code isn’t properly validated/tested.

10. Staying Grounded: How Leaders Should Navigate the AI Hype

• Data Over Hype:
  • Focus on measurement, experiment methodically, and combine developer workflow data with direct feedback.
  • “Data beats hype every time.” (Laura, 67:40)
  • AI is not a silver bullet; just giving licenses is insufficient—enablement and training are needed.

Notable Quotes and Memorable Moments

• On Hype:
  • “Can you imagine that headline: ‘ACME Corp only ships code to production that’s been read by robots. Is this the end of software engineering?’” (Gergely, 10:15)
• On Developer Experience vs. Productivity:
  • “Source code is a liability. …when what could have been written in one line is now written in five lines, do we really want to measure AI impact in terms of lines of code generated?” (Laura, 16:03)
• On the Reality of Time Savings:
  • “On the very best day, developers are not spending even 80% of their time coding. The average is like 25%...when we apply AI...we’re only working with 20% of that time.” (Laura, 30:08)
• On Satisfying Work:
  • “What happens with the time savings? ...develop[ers] were feeling less satisfied because AI is accelerating the parts that they enjoy.” (Laura, 00:04 & 30:08)
• On AI-powered Documentation:
  • “Documentation needs to be there for AI so that a developer gets the information they need in the best way…in the editor, not just your documentation site.” (Laura, 37:05)
• On Rollouts:
  • “The more intentional and structured a rollout is, the higher chance it has to be successful…slow is smooth and smooth is fast.” (Laura, 60:08)

Important Timestamps

• 00:00 – Opening question: what developer time savings with AI really mean
• 02:37 – How media hype misinforms and what engineering leaders need to know
• 09:14 – Why metrics like “acceptance rate” and “lines of code” are misleading
• 14:11 – What to actually measure: outcomes, not code volume
• 18:08 – Booking.com’s AI adoption strategy and performance data
• 24:04 – Biggest time-saving use cases for professional engineers (not code gen!)
• 30:08 – The paradox: AI accelerates “fun” parts, leaving more toil
• 36:17 – New architectural/documentation best practices for AI/agent usage
• 42:44 – Workhuman case study: 11% org-wide DX gain from AI
• 47:06 – Tooling investment, the return of big-ticket per-dev pricing
• 59:13 – Regulated industries’ advantage with structured AI rollouts (Indeed case)
• 63:51 – Automated migrations as a killer AI use case
• 67:40 – Laura’s closing advice: “Data beats hype every time.”

Takeaways for Engineering Leaders & Developers

• Measure, Don’t Guess: Build or baseline your own productivity and experience data before/after rolling out AI.
• Don’t Chase Vanity Metrics: Ignore lines of code and suggestion acceptance rates—focus on business/productivity outcomes.
• Adoption Requires Intentionality: Training and enablement matter as much as tool selection.
• Expect Complexity: AI doesn’t remove hard problems—batch size, testing, code quality still determine software health.
• Structured Experimentation Wins: Treat AI adoption as a series of experiments; regulated/structured orgs see the best outcomes.
• Focus on Developer Experience: Improving day-to-day dev workflows lead to real downstream benefits for teams and companies.
• Stay Skeptical, Stay Data-driven: Challenge hype with measurements; keep leadership and business partners informed with real numbers.

Recommended Resources & Closing

• Laura’s book recommendations:
  • “Write Useful Books” by Rob Fitzpatrick
  • “Unsavory Truth” by Marion Nestle
• Practical tip: For AI-assisted migrations, do one manual migration and use that as a diff example for the AI to repeat accurately.
• Link: For more data and guides, Laura recommends DX’s “Guide to AI-Assisted Engineering” (see DX website), and the Pragmatic Engineer Deep Dives.

“Data beats hype every time.” – Laura Tacho (67:40)

(End of summary)