The Stack Overflow Podcast

Episode: You need quality engineers to turn AI into ROI
Date: January 7, 2026
Host: Ryan Donovan
Guest: Pete Johnson, Field CTO of AI at MongoDB

Episode Overview

This episode dives into the crucial role of quality engineering and human expertise in making artificial intelligence (AI) initiatives deliver real return on investment (ROI). Host Ryan Donovan and guest Pete Johnson (MongoDB Field CTO) unpack a recent OpenAI “GDP VAL” paper, debate why so many AI projects fail, and share actionable guidance for developers and business leaders seeking to maximize value from AI systems. The conversation also explores MongoDB’s evolving toolset for AI-enhanced applications and reflects on the evolving responsibilities of software engineers in the age of advanced automation.

Key Discussion Points & Insights

Pete Johnson's Journey in Tech

• Began coding in 1981 as a sixth grader, turning a hobby into a multi-decade career.
• Long stints at HP (20 years, including early web app development and work on HP Cloud Services); other roles at Cisco, CDW, and now MongoDB.
• (02:45) "A lot has changed since the old TRS 80 days. Today everybody's talking about AI and agents..."

OpenAI’s GDP VAL Paper – How “Agentic” Tasks Are Measured

(03:20–07:04)

• OpenAI paper examined impact of LLMs across 44 occupations.
  • Tasks defined by experts with 14+ years’ experience.
  • Human experts and LLMs both attempted identical tasks; results judged “blind” by a third human evaluator.
  • AI-human collaboration saw the biggest productivity gains.
• Notably, OpenAI compared more than just their own models.
• Key stat: Highest-performing model still only "won" 47.6% of the time (Claude Opus), while GPT-4o lagged at 12.4%.

Memorable Quote:

"When the AI and the people work together... that's when they saw really big gains. ...It turned me into an AGI skeptic. I think we're entering an era where everybody's going to be AI-enhanced and see cost and speed improvements."
— Pete Johnson (06:07)

Why So Many AI Projects Fail

(07:15–09:58)

• Referenced MIT study: 95% of AI projects don’t deliver as promised.
• Two main reasons:
  1. No “SKU for AI”: Execs seek a packaged solution but undervalue organizational context and nuance.
  2. Flawed goal: Trying to replace people instead of augmenting them.
• Integration of AI to "enhance" people (rather than replace them) is more realistic and valuable.
• Injecting company-specific, proprietary content into LLMs (cost-effectively, without full retraining) is essential.

Memorable Quote:

"If you think you're going to get AGI and replace people, that's flawed logic... The real traction comes from improving the productivity of the people you have."
— Pete Johnson (07:28)

Human Skills Still Matter

(09:58–10:41)

• Like open source software, value comes not from the tool or data alone, but from people, process, and business insight.
• Recalled AWS re:Invent keynote (Werner Vogels) on the enduring importance of curiosity, communication, ownership, and “polymath” skills.

Turning AI into Measurable ROI

(10:55–12:26)

• Start with business problems, find which you have data for, and select those with measurable metrics.
• Good ROI comes from pairing strong business problems, proprietary data, and the right embeddings/vector search setup.
• MongoDB’s value-add: Offering developer-friendly tooling for vector search and high-quality embeddings (escalated further by Voyage AI acquisition).

Embeddings, Vector Search, and Abstraction Levels

(12:49–18:16)

• Five key decisions for developers working with embeddings:
  1. Choice of embedding model
  2. Similarity score (e.g., cosine similarity)
  3. Chunk size
  4. Number of dimensions
  5. Level of quantization
• Voyage AI acquisition enables distinctive features:
  • Matroska Reasoning: Lets developers experiment with lower-dimensionality embeddings (and storage reduction) without reprocessing corpora.
  • Contextualized Chunks: Embeds a sentence with its full-document context, yielding better retrieval quality with smaller chunk sizes and storage footprint.

Memorable Quote:

"We're still way closer... to the original JavaScript than to React or Angular. We want to move AI up those abstraction levels so more people can build, with less friction."
— Pete Johnson (17:11)

• The analogy: The AI development ecosystem is where JavaScript was pre-jQuery; MongoDB’s aim is to simplify and accelerate developer ramp-up.

Guidance for Developers Making Trade-Offs

(18:40–21:05)

• Default recommendations:
  • Similarity score: start with cosine similarity.
  • Chunk size: use contextualized chunking for smaller, more effective chunks.
  • Dimensions: begin with 1024, but easy to try 512 with Matroska/Matryoshka Reasoning.
  • Quantization: experiment with reduced precision for storage trade-offs.
  • Re-ranking: Using best-in-class re-rankers can provide an extra 10-15% boost in retrieval accuracy—vital for reducing hallucinations.

Indexing, Storage vs. Speed

(21:05–22:27)

• Index size correlates with retrieval speed; larger indexes can enable faster access but at greater cost.
• MongoDB Atlas (cloud service) offers automated, scalable vector indexing; users pick deployment region and scaling model.

The Place of MongoDB in Modern AI Workloads

(22:27–25:02)

• Origins: Founded post-cloud, mobile, and modern web era, focused on performance and flexibility (binary JSON/BSON storage, flexible schema).
• Unlike strict relational models (focused on minimal storage, rigid normalization), MongoDB enables thoughtful denormalization for AI-ready, low-latency applications.

Memorable Quote:

"The modern application—you can't have downtime on the weekend like you could in 1970... Slow is the new downtime."
— Pete Johnson (24:44)

Skills for the AI-Enhanced Era

(25:14–27:06)

• Werner Vogels' five themes for future engineers:
  1. Curiosity
  2. Communication skills
  3. Ownership of code (AI-generated or not)
  4. Systems-thinking
  5. Polymath/T-shaped expertise

Memorable Quote:

"Just because you might use AI enhanced tooling to generate your code, you still own it, you're still responsible for it running in production."
— Pete Johnson (25:38)

AI Code Is Like All Code: Know What You Ship

(27:06–27:33)

• “Can you trust AI code? No. Can you trust junior developer code? No. Can you trust code you wrote yesterday? No. Make sure you look at and understand any piece of code that comes across your desk.” — Ryan Donovan

• The key is having effective review cycles and not blindly trusting AI-generated output.

Memorable Quote:

"I haven't written a manual line of code in eight months now."
— Pete Johnson (27:30)

Notable Quotes

• On AI-Human Synergy: “AI with an expert is just tons better.” (07:04, Ryan Donovan)
• On What Matters Most: “It's not the software that is the special sauce, right? It's the business, it's the people.” (09:58, Ryan Donovan)
• On the Evolution of Tools: “Lowering that learning curve and reducing that friction for the individual developer is a really big part of [our work].” (17:23, Pete Johnson)

Segment Timestamps

• 00:47 — Pete’s background and tech journey
• 03:20 — The design and findings of OpenAI’s GDP VAL paper
• 07:15 — Reasons behind 95% AI project failure rate
• 10:55 — How to actually realize ROI from AI projects
• 12:49 — Technical dive into vector search, embeddings, and MongoDB’s differentiation
• 18:40 — Practical recommendations for developers making AI trade-offs
• 22:27 — Positioning MongoDB in the modern application landscape
• 25:14 — Future-facing skills for AI-enhanced software engineering

Takeaways for Developers & Leaders

• AI is not a silver bullet—ROI requires specific business problem targeting, strong proprietary data, and deep integration of human expertise.
• Enhancing workers and augmenting productivity should be the focus, not replacement.
• Advances in embeddings and vector search (like contextualized chunks and matroska reasoning) are making sophisticated retrieval and customization easier for developers.
• Building and maintaining AI systems demands the same ownership, curiosity, and communication skills as classical engineering—now with the added complexity of “reviewing” (not just writing) code.
• The next three years will see an explosion in agentic, AI-enhanced software. Lowering the entry barrier for developers is critical.

Final Thoughts

This episode argues convincingly that actualizing AI’s business potential depends as much on quality engineering, solid metrics, and human expertise as on the models themselves. “AI-enhanced” will soon be the norm—not a replacement, but an amplifier for human creativity, judgment, and communication.

For more details on technical questions, MongoDB’s AI offerings, or Pete Johnson’s recommended resources (including Werner Vogels’ AWS re:Invent keynote), see the show notes.