Podcast Summary: The Stack Overflow Podcast

Episode: Every ecommerce hero needs a Sidekick

Date: January 9, 2026
Host: Ryan Donovan
Guest: Vanessa Lee, VP of Product at Shopify

Episode Overview

This episode explores how AI is transforming the world of ecommerce, focusing on Shopify’s innovative AI assistant, Sidekick. Ryan Donovan sits down with Vanessa Lee, VP of Product at Shopify, to discuss AI as a "renaissance" in technology, the evolution of Shopify’s developer platform, and how grounded evaluation methods ensure safe and helpful AI products. They also dive into the technical and human challenges of training AI, how Sidekick is reshaping merchant workflows, and share insights on the future of customizable, AI-driven interfaces and developer tooling.

Key Discussion Points & Insights

1. Vanessa’s Journey into Software and Shopify

• Background: Vanessa Lee started as a robotics (mechatronics) engineer working with C and Java on early hardware projects (01:00).
• Transition to Shopify: Joined Shopify nine years ago, when the platform was nascent; describes early days without API versioning or robust extensions (01:00–02:59).
  • Vanessa: “We didn't have, we didn't version our APIs when I joined, which was wild … it's amazing to see ... how much our platform has grown.” (02:34)

2. AI as a Renaissance in Technology

• AI’s Prominence: Discussed the skepticism around AI and how, paradoxically, the more developers use it, the more skeptical they become (04:12).
• Shopify’s Ambition with Sidekick: Originated in 2024 to be a true productivity partner for merchants (04:29).
  • Vanessa: “It showed Sidekick … able to draft a whole bunch of resources in Shopify. That was really the start of our Sidekick journey ... The last couple of years have really been an exercise of how do you build an AI agent at scale.” (04:29–05:43)

3. What is Sidekick?

• Definition: Shopify’s AI assistant, embedded in the UI, helping merchants create and manage products, collections, and store settings (05:43–06:12).
  • Vanessa: “It lives alongside in the UI. You can ask it questions, it could create products for you, can create collections for you ... it can help you edit your online store.” (05:46)

4. Ensuring AI Quality – Data, Evaluation, and Human Judgment

• Preventing Hallucinations: Importance of rigorous evaluation sets and using LLMs to grade other LLMs; negative test cases highlight poor behaviors (07:15–08:34).
  • Vanessa: “You use LLMs to generate synthetic data to form ground truth sets … it’s a lot of grunt work, it’s a lot of time investment, but it’s also being super creative about how we build this dataset of evaluations.” (07:15)
• Humans in the Loop: Human editors are crucial to setting the ‘spec’ for Sidekick; humans refresh the ground truth sets monthly (08:54–14:29).
  • Vanessa: “If you are building AI features, funnel all of your opinions … into ground truth set. That's now the new spec for building AI and that's the human in the loop.” (10:24)
• Evaluations as Specification: Evals are the new requirements documents in AI development (08:54–11:09).

5. The Dangers of Synthetic Data and Model Collapse

• Synthetic vs. Real Data: Initially relied heavily on synthetic data, but now mostly use real user conversations to grade performance as Sidekick’s adoption grows (12:33–13:21).
  • Vanessa: “It’s actually not that easy to generate synthetic tests … now that we’ve actually gone live … we use [real conversations] more than synthetic.” (13:05)

6. Sidekick and UI Customization

• Beyond Text Assistants: Shopify is experimenting with Sidekick building custom UI and apps tailored to merchant needs—personalized interfaces for unique workflows (14:51–19:05).
  • Vanessa: “One of the things … is how does Sidekick come out of just a text shell? … how does LLM help to change the way we interact with UI as well?” (14:51)
  • “We’re launching … the ability for Sidekick to generate apps for you, custom applications for your business.” (16:56)
• Personalized Experiences: AI-generated tools move beyond code-oriented users to empower non-technical merchants (17:13).

7. Product Data and Categorization Challenges

• Standardizing Product Data: Shopify uses LLMs to auto-categorize products and suggest attributes, enhancing discoverability and integration with partners like OpenAI (19:19–21:56).
  • Vanessa: “If you started and said, ‘Okay, I'm creating a new product’, … an LLM would run … we have a standardized taxonomy … [then] automatically suggested for you.” (20:00)
• Continuous Taxonomy Expansion: Robust and continually growing taxonomy is necessary due to diversity in products and seller data (22:13).

8. Balancing Developer and Novice User Needs

• Shopify aims to empower both code-savvy developers (via headless architectures and APIs) and non-technical merchants (via no-code tools and theme customization) (23:22–25:42).
  • Vanessa: “We always underestimate how much merchants care about their brand and how much it’s about expression.” (23:32)

9. The Future of Sidekick and AI-Driven Ecosystem

• Third-Party Integrations: Introduction of ‘app intents’ to let Sidekick interact with ecosystem apps, opening more developer opportunities (25:51–27:08).
  • Vanessa: “One of the things that we started releasing … is our ability for Sidekick to essentially launch what we call app intents, … ways for you to register tools for Sidekick … so that merchants can actually access your app from conversations.” (26:08)
• Developer Tools Upgrades: The MCP tool and Shopify CLI upgrades streamline app development and deployment for third parties (27:15–28:33).

Notable Quotes & Memorable Moments

• On the AI Renaissance:
  “We put in a lot of hard work to get to this point and to make AI something that wasn't just a great demo feature, but something that actually repeatedly would deliver value.” — Vanessa Lee (06:33)
• On Human Oversight:
  “The human in the loop is what really raises the bar. And to be honest, that work never really ends.” — Vanessa Lee (14:23)
• On Expanding Customization:
  “How do you give that power to a user that’s not as technical? That is … something that we would not have been able to do without AI.” — Vanessa Lee (17:13)
• On Empowering Developers and Merchants:
  “We want to be with you no matter if you choose to develop your own, let’s say headless storefront or if you are coming in and installing a theme.” — Vanessa Lee (25:04)
• Looking Ahead:
  “One of the questions that I get a lot from our ecosystem is … how can Sidekick then go create the discount … and let me draft that in an email. … One of the things we started releasing … is our ability for Sidekick to … launch app intents.” — Vanessa Lee (26:08)

Timestamps for Important Segments

• 01:00–02:59 – Vanessa’s background & path to Shopify
• 04:29–06:12 – What is Sidekick? How Shopify’s AI Assistant Works
• 07:15–10:24 – Preventing hallucinations, grounding Sidekick in robust evaluations
• 11:09–14:29 – Evaluations as specs; importance of the human in the loop
• 14:51–19:05 – Future vision: Sidekick for UI customization and auto-generated apps
• 19:19–22:13 – AI-powered product categorization and data challenges
• 23:22–25:42 – Serving both developers and non-technical merchants
• 25:51–28:33 – Opening Sidekick to app integrations and discussion of new developer tools

Tone and Style

The discussion is pragmatic, transparent, and focused on real-world developer and merchant needs. Vanessa is candid about the technical and human challenges behind building and scaling Sidekick, while the overall tone remains optimistic and forward-looking about AI’s potential in ecommerce. The technical complexity is grounded with practical examples and relatable anecdotes.

For More Information

• Find the episode and additional links in the show notes.
• Follow Ryan Donovan (host) on LinkedIn; Vanessa Lee is on X (@vlaurenlee).