Stream's AI support agent resolves 80% of their customer tickets. It was built by one engineer, working part-time, inside a 60-person product and engineering team.

In hundreds of conversations with CX leaders evaluating self-build, this is the exception. Most attempts stall between an impressive hackathon demo and a system reliable enough for production. Nick Rogers, Chief Product and Technology Officer at stream (formerly Wagestream), explains why his team is one of the few that crossed that gap.

Nick walks through the test he'd give any leader weighing build versus buy (his answer: if you need to ask, partner), the "wait for the frontier" tempo of trying, failing, and coming back when a new model release shifts what's possible, and the results of stream's recent model bake-off, including the model that surprised the team for being worse than expected.

Follow The Squawk podcast for hot takes and big ideas in CX. Perfect for a short commute and guaranteed to leave you with something new to think about, or implement, for your CX operation.

Timestamps:

00:00 - Cold open

00:41 - Intro to Nick & stream

01:55 - Nick's career path

04:00 - Why stream's self-build is the exception

05:07 - The chronological story: GPT-3.5 to Gemini 2.0 Flash

07:50 - The culture that lets stream try and fail

09:15 - Eval methodology: golden datasets and A/B testing in production

11:30 - 70% to 80% deflection, a 33% reduction in tickets

11:55 - The model bake-off and the Claude Opus surprise

13:20 - Latency versus quality tradeoffs

14:46 - "How many of your 60 engineers work on this?" - effectively one

16:18 - Why talent matters (Joel on Software's "high notes")

17:39 - The self-build vs buy test: "if you need to ask, just buy"

19:30 - The "wait for the frontier" strategy

20:58 - Risk: from wild hallucination to agentic actions

24:41 - Knowledge management as the new CX job

26:42 - Lightning round

28:51 - Recommended reading: Hackers and Painters by Paul Graham

29:48 - Wrap-up

Where to find Nick:

LinkedIn: https://www.linkedin.com/in/rogersnm/