Product Thinking Podcast: Episode 255

How Shopify Is Leveraging AI at Scale with Vanessa Lee

Host: Melissa Perri
Guest: Vanessa Lee, VP of Product at Shopify
Date: November 5, 2025

Episode Overview

In this episode, Melissa Perri speaks with Vanessa Lee, Shopify’s VP of Product, about Shopify’s journey in integrating AI at scale, specifically around the development of Sidekick, their agentic commerce assistant. They delve into the emerging paradigm of “agentic commerce,” discuss the technical and organizational challenges of building AI-powered tools for millions of merchants, and explore what the future holds for commerce and product management in the era of large language models (LLMs). The conversation is candid, insightful, and full of practical lessons for product leaders navigating AI transformation.

Key Discussion Points & Insights

Vanessa’s Journey at Shopify (02:46–03:50)

• Vanessa joined Shopify as a Senior Product Manager and rose through the ranks over nine years by working across developer ecosystems, the online store, and AI initiatives.
• Emphasizes Shopify’s unique, hands-on product management culture:

  “We are the type of product managers who just absolutely love building and being in the details... our team is actually quite light when you look at ratios between PM and engineering.” (04:02, Vanessa Lee)

The Vision for AI at Shopify: Sidekick (05:18–10:17)

• Why Sidekick? Shopify’s core mission is empowering entrepreneurs. Entrepreneurship is a lonely journey, so Sidekick was envisioned as an always-available, business-savvy digital assistant.
• Capabilities: Sidekick isn’t just a chatbot—it helps with everything from creating products/collections and setting up payment methods to providing analytics-driven insights and growth advice.
• Real Value: Merchants use Sidekick for higher-order tasks—trend analyses, sales forecasting, marketing strategy, etc.

  “Merchants use us a lot for higher order tasks, which was also a surprise... like a data analyst, an advisor, a financial advisor understanding what marketing tactics they should explore.” (09:17, Vanessa Lee)

Building and Iterating Sidekick: An AI PM's Challenge (11:01–21:50)

• Early Struggles: The conversational domain is “the easiest thing to demo and the hardest thing to build in production” because of high variability and undirected user inputs.

  “When it comes to conversation, the user could ask anything and it is so difficult to succeed and provide good answers across all the topics that could be asked.” (11:32, Vanessa Lee)

• Synthetic Data for Evaluation:
  • Used LLMs to generate thousands of simulated merchant queries and synthetic conversations.
  • Human PMs graded these for quality, and their judgments were used to train an “LLM judge” to automate conversation grading.
• LLM Judge as Living Spec:
  • The LLM judge grades outputs according to product values (accuracy, safety, tone, tool use, etc.), becoming a foundational tool for ongoing development and live monitoring.
  • Developers run all changes against this judge, making it a “living, ever-evolving spec.”

  “This LLM judge becomes the tool that you use to guide the team. It’s so much more than a spec... it becomes the measurement of whether or not we are achieving what we want as a product team in this agent.” (19:29, Vanessa Lee)

Practical Advice for PMs Building with AI (22:51–25:47)

• Get Creative: PMs should actively experiment and play with AI—beyond chat, use it for label generation, data cleaning, etc.
• Applications at Shopify: LLMs help auto-categorize products, suggest attributes (e.g., length of a garden hose), and make ecommerce experiences more structured for various marketplaces.
• Lesson: Leverage LLMs as flexible data tools, not just end-user assistants.

Limitations & Pervasiveness of LLMs (25:54–26:38)

• LLMs are now embedded into everything at Shopify; finding a place where they aren’t useful is harder than finding good applications.
• Internal adoption is widespread—executives use AI assistants for deep dives and rapid prototyping, with no internal stigma about leveraging AI.

Scaling AI across Shopify (27:03–28:24)

• Shopify’s strength: Curiosity-driven, risk-embracing culture from leadership down.
• Being early and bullish on emerging tech (AR/VR, QR codes, stablecoin, and now AI) enables Shopify to deliver cutting-edge merchant tools.

The Agentic Commerce Future (28:42–33:08)

• Definition: Agentic commerce is the integration of agents (digital assistants) to transform product discovery, purchasing, and execution.
• Vision: Assistants will handle not only finding products but also completing transactions (including logistics like checkout and delivery).
• Shopify’s Edge: Modular, checkout experiences for millions; investment in taxonomy and data models means merchants’ products are ready for agent-driven discovery and purchase.
• Checkout As a Platform: Shopify invested in a modular, extensible checkout system that powers both small stores and large brands equally.

Democratizing Commerce for all Merchants (33:08–36:38)

• Both small, scrappy entrepreneurs and large enterprises benefit—smaller merchants use Sidekick holistically, from creation to marketing; big brands use it in focused, role-based ways.
• Larger merchant executives are also starting to use AI more holistically—e.g., brainstorming optimization strategies.

Partnership with OpenAI & The Commerce Protocol (36:38–38:07)

• Shopify and OpenAI have partnered to enable product discovery and end-to-end commerce directly in ChatGPT, making every merchant’s store accessible through conversational commerce.
• The integration eliminates multi-platform friction for consumers and expands reach for merchants.

The Evolution of Marketing and Commerce in an AI World (38:07–40:59)

• Merchant focus is shifting to ensuring clean, rich product data and well-managed metadata so agents can accurately represent products in natural conversation.
• Shopify invests in merchant tools for data quality, e.g., the Knowledge Base app, allowing merchants to see and proactively answer common questions.
• Platform power: “One integration” from Shopify enables millions of merchants to immediately surface in AI-powered discovery, search, and commerce.

The Future of Product Management in AI (40:59–43:36)

• PMs are now empowered to prototype, tinker, and run “vibe coding interviews”—bridging the gap between idea and implementation, reducing dependence on dev bandwidth.

• “PMs can now prototype stuff that we would typically have to rely on the dev team being free and we can just tinker with it... that process, that feedback loop is so tight.” (41:09, Vanessa Lee)

• The discipline is changing—communication, prototyping, and rapid iteration are the new superpowers.

Notable Quotes & Memorable Moments

• On AI product eval:

  “You’re basically imparting all of your product intuition into an LLM to then shape yet another LLM, which is so mind boggling... This is the type of creativity you need.” (20:06, Vanessa Lee)

• On workplace culture:

  “There’s no ego in Shopify about not using AI to help us do our jobs. That’s for sure.” (26:38, Vanessa Lee)

• On PM career advice:

  “Instead of chasing titles or like particular companies, think about how you want to spend your time and then go find the place where you’re going to get to do that.” (43:43, Vanessa Lee)

• On the future for SMBs:

  “We always try and make sure that we are democratizing access to customers and helping all entrepreneurs succeed, whether it’s a small merchant or a really large merchant.” (33:27, Vanessa Lee)

Important Timestamps

• 02:46 – Vanessa’s journey at Shopify
• 05:18 – The vision behind Sidekick and agentic commerce
• 07:35 – Real-world use cases for Sidekick
• 11:01 – Story of building and evaluating Sidekick with LLMs
• 18:31 – LLM judge as a living spec for AI product development
• 22:51 – Practical advice for PMs building with AI
• 25:54 – LLMs’ expanding role and limitations
• 27:03 – Shopify’s culture and scaling AI
• 28:42 – The definition and future of agentic commerce
• 33:08 – Serving SMBs vs. large brands with AI
• 36:38 – Shopify’s partnership with OpenAI and ChatGPT integration
• 38:07 – The future of marketing via AI-driven agents
• 40:59 – The changing toolkit for product managers
• 43:43 – Advice for aspiring PMs

Final Takeaways

• Agentic commerce and AI are fundamentally reshaping the ecommerce landscape.
• Shopify’s approach is deeply merchant-centric, democratizing access to the future of commerce for businesses of all sizes.
• The culture of curiosity, rapid prototyping, and embracing cutting-edge technologies gives Shopify a crucial competitive edge.
• Future product managers must combine curiosity, technical literacy, and prototyping skills to drive impactful products in the age of AI.

For listeners looking to dive deeper, follow Vanessa Lee on X (Twitter), check out Shopify’s agentic commerce features, or visit the Product Thinking Podcast website for further resources.