Podcast Summary: Always Be Testing #102

Episode Title: Building High-Performing Teams & Smarter Testing with John Toskey
Host: Tye DeGrange
Guest: John Toskey
Date: September 30, 2025

Episode Overview

This episode brings together host Tye DeGrange and industry veteran John Toskey for a thoughtful and energizing discussion on building high-performing teams, cultivating a culture of experimentation, and navigating the ever-evolving world of performance marketing, affiliate, and retail media. The conversation delivers actionable insights for marketing leaders, team builders, and data-driven growth professionals, while mixing in personal anecdotes, trends, and even a glimpse into John's personal hobbies.

Key Discussion Points & Insights

1. Building & Scaling High-Performing Teams

• Diversity in Teams:
  John emphasizes the importance of true diversity—not just in gender or ethnicity, but also in geography, experience levels, and backgrounds.
  • “Having enough people in different locations, having enough of a breadth of experience and levels in the team… there’s a longevity to how the team can perform and grow.” — John Toskey [01:28]
• Managing Growth & Opportunity:
  Teams stagnate when everyone stays put and growth opportunities dry up, leading to a “log jam.” John advises a constant influx of junior talent to keep the team dynamic, which also creates a natural pipeline for future leaders.
• Promoting Employee Growth:
  • “I have to be willing to set them free and advocate for them someplace else to keep them at the company.” — John Toskey [04:26]
  • Maintaining positive alumni who promote your company, even after they leave, is seen as a critical leadership responsibility.

2. Creating a “Test and Learn” Culture

• DNA of Experimentation:
  Companies with a true "testing culture" make it obvious from the inside out; it’s natural, encouraged, and systemic. [05:50]
• Socializing Test Learnings:
  A common pitfall: test results aren’t communicated beyond the immediate team or up/down the org chart.
  • “One of the biggest sins of companies is there’s so much testing going on…and it won’t get socialized beyond that team.” — John Toskey [06:41]
• Institutional Knowledge Loss:
  Without properly sharing and documenting experiments, companies end up retesting old ideas or fixing issues already solved.
  • “You realize you’re testing something that somebody tested three years ago…or they don’t know, which is more likely.” — John Toskey [07:59]
• Aligning Testing with Strategy:
  Too often, tests answer the wrong questions or don’t align with impactful strategies:
  • “Whoever’s proposing the experiment isn’t actually answering the right question that they want to…you might have a question…[but] the significance of the test isn’t big enough to ultimately impact where you’re trying to get to.” — John Toskey [08:40]
• Avoiding Small Thinking:
  Both host and guest warn against thinking too small with experiments, especially when business objectives require larger leaps. [09:36]

3. Moving Beyond Statistically Significant (Statsig) Thinking

• Limitations of Statsig:
  Companies can become paralyzed waiting for statistical significance, missing out on actionable, directionally correct insights.
  • “At a certain point, can you just call it and say directionally this is good enough? And I don’t think that probably happens as much as it needs to.” — John Toskey [10:29]
• Balancing Data & Intuition:
  While data usually trumps intuition, John points out that underlying assumptions in the data and analysis need scrutiny.
  • “At some point, somebody, probably somebody fairly junior, has made an assumption someplace…sometimes those assumptions are inconsequential. Sometimes they’re hugely consequent.” — John Toskey [11:54]
• The Power of Documentation:
  Keeping track of why a test was run and the original hypotheses enables teams to revisit, adapt, and avoid repeated mistakes. [12:59]
• Relationship with Finance:
  Strong alignment with finance ensures everyone is judging test results from the same baseline, especially around metrics like CAC and LTV.
  • “...being able to storytell to who holds the purse strings and build relationships so that you have credibility with that organization…” — John Toskey [16:27]

4. Trends & Future of Performance Marketing, Affiliate, and Retail Media

• Unpredictability & Opportunity:
  The only constant in marketing is change; sometimes new channels (e.g., Buy Now Pay Later) rise unexpectedly. [17:06]
• Affiliate vs. Brand/Influencer:
  Ongoing debates about whether influencer/creator channels belong in “performance” or “brand” marketing; lack of clarity on effective attribution.
  • “Marketers need to separate out their lower funnel affiliate programs with their upper funnel branding and creator because…is it truly performance marketing or is it more brand marketing?” — John Toskey [18:33]
• Rise of Retail Media Networks:
  Especially pronounced among enterprise retailers, but smaller marketers are still figuring out how to leverage first-party data for offsite targeting and partner deals.
  • “If somebody can solve that, that would be huge…but those are the same problems we’ve had for years…” — John Toskey [19:47]
• Cyclical Nature of Budgeting:
  Brands like Nike and Airbnb swing between bottom-funnel (performance) and top-funnel (brand) spend, seeking a mythical perfect balance.
  • “There’s no right answer, but it’s really challenging if you’re a CFO or CMO trying to understand how it fits together and look at things apples to apples. Because they’re not ultimately and everybody wants a one to one explanation of everything.” — John Toskey [21:43]

5. The AI Surge: Hopes, Fears & the Skills Gap

• Excitement & Uncertainty:
  There’s a flood of investment in AI products, but a real risk of wasted effort as the industry sorts itself out.
• Generational Skills Gap:
  John notes stark differences in how younger vs. older employees leverage AI tools, as well as differing company comfort levels in feeding proprietary data to AI.
  • “Right now you have this conflict of everybody in the company should be using AI. God, don’t put that in. So how do I use this as a tool? … There’s a big culture shock that’s going to hit us here in the next few years.” — John Toskey [25:17]
• Not Just Evolution—A Step Change:
  Unlike the adoption of computers or the internet, generative AI hands more agency to the tool itself, not just the user.
  • “...It’s not access to information. It’s not access to something that can accelerate your own output. It’s potentially access. It’s generative. Like, the internet’s not generative.” — John Toskey [27:00]

Notable Quotes & Memorable Moments

• On releasing great talent:
  “If somebody’s really, really good, I have to be willing to set them free and advocate for them someplace else to keep them at the company.” — John Toskey [04:26]
• On testing cycles:
  “You realize you’re testing something that somebody tested three years ago. And maybe it’s because the person who wants to test it now doesn’t believe what they saw before. That can’t possibly…right? There must have been something wrong in the test setup. Or they don’t know. Which is more likely.” — John Toskey [07:59]
• On generative AI:
  “You're handing controls and output and agency to something else that you don't necessarily control.” — Tye DeGrange [27:12]
• On using AI at work:
  “Right now you have this conflict of everybody in the company should be using AI. God, don’t put that in. So how do I use this as a tool? And how do you then train this dichotomy of older employees and younger employees so that you have policies and language that support both?” — John Toskey [25:17]
• On balancing data and intuition:
  “Data always wins, right? Because especially intuition is typically one person. Maybe you can get some consensus around an intuition, but for me, a lot of it goes back into the assumptions that are being made within the test.” — John Toskey [11:48]
• On work-life balance and self-discovery:
  “I love to cook and as I've gotten older I love to cook more…It's something that I used to feel like was another chore and I think I've gotten more zen about it.” — John Toskey [29:35]

Personal & Fun Segments

Favorite Software & Products

• Merlin Bird ID by Cornell (birdsong identification app)
  Both guest and host share their nerdy appreciation for this “epic” app. [27:56]
• Dr. Tung’s Smart Floss:
  John passionately recommends it as a unique and highly effective dental hygiene product. [28:32]

Personal Passions

• Cooking:
  John is an avid home cook (vegan), preparing most of his family’s meals and even shopping across multiple grocery stores for fresh ingredients. [29:35–30:48]
• Favorite Vegan Dish Recommendation:
  For non-vegans: a hearty lentil or bean soup as an approachable introduction. [31:16]

Timestamps for Key Segments

• Building High-Performing Teams: [01:17–05:35]
• Test & Learn Culture: [05:35–09:59]
• Statsig vs. Directional Data and Assumptions: [09:59–12:59]
• Relationship with Finance: [15:07–16:52]
• Performance Marketing/Affiliate Trends: [17:06–21:30]
• Retail Media Networks: [18:56–20:23]
• Brand vs. Performance Budgeting: [21:30–22:19]
• AI in Marketing & Generational Shift: [22:37–27:18]
• Personal Software Favorites: [27:46–28:13]
• Consumer Product Favorite: [28:32–29:24]
• Cooking & Vegan Food Chat: [29:35–31:42]

How to Connect with John Toskey

• Email: toskymail@gmail.com
• LinkedIn: [Redirect via toschi.com]

Episode Tone & Language

Warm, conversational, occasionally self-deprecating, and replete with practical wisdom and humility. John’s language is thoughtful and direct, while Tye brings enthusiasm and relates guest insights back to listener experience.

This episode is a treasure trove for marketers, team leaders, and anyone curious about marrying experimentation with sustainable team dynamics and real business growth—plus, you’ll leave with a soup recipe idea and a new floss favorite.