Podcast Summary: The Analytics Power Hour #278

Title: Is AI Good at Data Analysis? That's the Wrong Question?
Date: August 19, 2025
Hosts: Michael Helbling, Moe Kiss, Tim Wilson, Val Kroll, Julie Hoyer
Guest: Juliana Jackson (Associate Director of Data and Digital Experience at Monks; Co-host of Standard Deviations Podcast; Author of "Beyond the Mean" newsletter)

Overview

This episode explores a widely debated question in the analytics community: Is AI (specifically Large Language Models or LLMs) actually good at data analysis—and is that even the right question to be asking? With special guest Juliana Jackson, the hosts dig into the realities and misconceptions about how AI is being used (and hyped) in data analytics, the industry’s pressure to adopt AI, and what analysts should really focus on as the field evolves.

Key Discussion Points & Insights

1. The Origin of Juliana's AI Analysis Rant

[02:42] – [06:40]

• Juliana shares the story of starting the "Standard Deviations" podcast and how Simo Ahava became her co-host.
• Both hosts and guest express frustration with the oversimplification of analyst work, especially when people claim LLMs can replace genuine data analysis.
• Quote (Juliana, 08:26):
  "I keep on seeing a lot of people on LinkedIn dumping a lot of [data] into ChatGPT... and they're kind of like minimalizing the work an analyst does and how much experience does it actually take to be able to look at numbers and draw conclusions."

2. LLMs and “Probabilistic Uncertainty” in Analysis

[13:48] – [16:52]

• Discussion about the fundamental differences between LLMs (probabilistic outputs) and classic statistical analysis (deterministic).
• LLMs generate different responses to the same prompt, while real statistical methods are consistent and reproducible.
• Quote (Tim, 15:02):
  "If I put a prompt into ChatGPT and immediately put the exact same prompt in, I'm going to get a different response. ...Uncertainty is built into them, which is unnecessary for the task of crunching numbers."

3. The Real Root of Pressure: VC Hype and Industry FOMO

[18:32] – [22:21]

• Juliana argues that much of the push for "AI everywhere" stems from VC-driven tech hype, not actual business needs.

• AI is often deployed as a feature to attract funding or market attention, not necessarily because it's beneficial or needed for the task.

  Quote (Juliana, 18:32):
  "The actual problem is the VC bullshit that's happening... I wouldn't need an LLM on my CRO tool to think about the hypothesis. Why? Just let me do my A/B testing in peace."

• Pressure on analysts and marketers to “innovate at all costs” is making many feel vulnerable and uncertain about their career value.

4. The Pitfalls of “AI = LLM = Magic” Thinking

[24:14] – [25:14]

• AI has become synonymous with ChatGPT or LLMs, ignoring the wider field (like programmatic ads, predictive modeling, etc.).

• This is a dangerous and narrowing perception in the industry.

  Quote (Tim, 24:14):
  "There's a more dangerous narrowing of, 'Oh, AI equals LLM equals chat interface for anything.' That's an unfairly narrow definition."

5. Self-Service Analytics, Chat Interfaces, and Business Abstractions

[25:14] – [30:09]

• There's a renewed industry push for self-service analytics and chat-based interfaces as the next big solution.
• While abstraction makes things appear simpler to business users, genuine complexity in business data remains; not everything can be (or should be) automated.
• There’s a risk that LLMs, eager to offer answers, won’t probe for necessary details—missing business context and nuance.
• Quote (Val, 25:14):
  "The business will have an easier way to deal with an abstraction than with statistical methods... I do understand the need for abstracting a lot of what we do."

6. Anthropomorphizing and Emotional Attachment to AI Tools

[28:53] – [32:48]

• Analysts (and people in general) tend to anthropomorphize AI tools, which can create comfort and attachment but also over-trust.

• LLMs are seen not just as tools, but as “colleagues” or sounding boards—a throwback to early chatbots like ELIZA.

  Quote (Juliana, 32:48):
  "If we take it to the analytics world, it's very nice… to be able to talk to somebody that gets it… without you feeling that you're an idiot. It's a tale as old as time, like we are the problem."

7. The State of Analytics Community and Vendor Hype

[34:36] – [37:24]

• Juliana laments the decline in vendor-sponsored education and honest thought leadership.
• Vendors, driven by hype rather than substance, are “taking advantage of blurry lines” in the market to sell AI tools that may not actually help analysts.
• Quote (Juliana, 36:58):
  "Automation can scale outcomes, but not competence, and AI cannot fix broken value propositions for vendors. If users didn't come before AI, they're not going to come after AI."

8. AI as an Amplifier, Not a Shortcut

[37:24] – [41:48]

• AI can help juniors or accelerate certain tasks for experts, but it doesn't replace foundational knowledge, context, or fair critical thinking.

• Moving from “zero to one” in skill may be impressive, but “one to two” (or expert level) requires deeper learning still not matched by AI.

• LLMs can be a shortcut, but not a substitute for understanding data, business context, or analytics methodology.

  Quote (Michael, 39:10):
  "LLMs are prediction engines... they're usually giving you some average that gets you further than you would get yourself… But I'm not an excellent developer with AI. An excellent developer is thinking about all kinds of other things... The AI is not thinking about any of that."

9. Multimodal AI & Practical Recommendations

[41:48] – [47:09]

• Discussion on the promise of multimodal AI (combining language, visual, and other modes) for future analytics.
• Juliana strongly recommends using the right-sized model for the job—small language models for specific tasks, LLMs for exploratory or ideation tasks.
• LLMs should be used for discovery or proof of concept, but for production-level analysis (especially unstructured data), smaller, more controllable models are preferable.
• Quote (Juliana, 42:02):
  "Language models are good for language... I will never choose to use an LLM to do any type of unstructured data analysis. I will always go for a small language model... because you have more control."

Notable Quotes (with Timestamps)

• "I keep on seeing a lot of people on LinkedIn dumping a lot of [data] into ChatGPT... and they're kind of like minimalizing the work an analyst does..." — Juliana Jackson [08:26]
• "If I put a prompt into ChatGPT and immediately put the exact same prompt in, I'm going to get a different response. ...Uncertainty is built into them, which is unnecessary for the task of crunching numbers." — Tim Wilson [15:02]
• "VC bullshit... I wouldn't need an LLM on my CRO tool to think about the hypothesis. Why? Just let me do my A/B testing." — Juliana Jackson [18:32]
• "There's a more dangerous narrowing of, 'Oh, AI equals LLM equals chat interface for anything.' That's an unfairly narrow definition." — Tim Wilson [24:14]
• "Automation can scale outcomes, but not competence, and AI cannot fix broken value. If users didn't come before AI, they're not going to come after AI." — Juliana Jackson [36:58]
• "Language models are good for language... I will never choose to use an LLM to do any type of unstructured data analysis." — Juliana Jackson [42:02]

Memorable Moments

• Juliana’s self-deprecating humor and honesty about feeling "lazy" yet obsessive about proper methodologies.
• Debate over why analyst jobs matter and shouldn’t be replaced by shallow AI promises.
• Multiple hosts openly sharing industry anxieties about keeping skills relevant in the age of AI.
• Juicy inside stories about podcasting in the analytics industry.

Practical Takeaways

• Don’t conflate “AI” with “LLM” or “chat interfaces”—AI is a broad field.
• LLMs are great for exploration, ideation, or for those with some knowledge. For production analysis: use statistical/machine learning tools.
• Business pressure to “innovate with AI” may be more about trends than actual business value; analysts should focus on fundamentals and critical thinking.
• Vendors and organizations should invest more in genuine education, not just feeding the latest hype.
• Use small language models for specific, repeatable analytics tasks—you get more control and reliability.

Timestamps for Key Segments

• Intro and Welcome: [00:00] – [02:16]
• Juliana’s Podcast Origin Story: [02:16] – [06:40]
• AI/LLMs and Data Analysis: What’s Missing?: [06:40] – [13:48]
• Probabilistic Pitfalls of LLMs: [13:48] – [16:52]
• Why is the Industry Chasing LLMs?: [16:52] – [24:14]
• Democratization, Chat UIs, and Business Users: [24:14] – [30:09]
• Human Attachment to AI Tools (Anthropomorphism): [30:09] – [32:48]
• Vendor Hype and Lost Thought Leadership: [34:36] – [37:24]
• AI as Amplifier, Not Short-cut: [37:24] – [41:48]
• Multimodal AI & Model Selection Tips: [41:48] – [47:09]
• Wrap-Up & Last Calls: [47:09] – [56:25]

Hosts' & Guest’s Tone

• Candid, sharp, and irreverent: No one shies from critique, and Juliana’s blend of humor and bluntness keeps things lively.
• Reflective yet practical: Numerous admissions of industry anxiety but always grounded in actionable advice and appreciation for foundational skills.

For Further Exploration

• Juliana Jackson’s “Beyond the Mean” newsletter – Deep dives, rants, and actual research about analytics and AI.
• Standard Deviations Podcast – Co-hosted by Juliana and Simo Ahava, focusing on analytics and digital experience topics.
• TED Talk referenced: "This is What a Digital Coup Looks Like" by Carol Cadwalladr [47:57]
• MeasureCamp Chicago Event and the Ologies Podcast (mentioned in Last Calls)

Bottom Line:
Rather than ask “Is AI good at data analysis?”—the real questions are WHAT problem are you solving, WHAT tool or model is best for that job, and HOW are you ensuring critical thinking, context, and community? LLMs are powerful, but they’re not magic—and the future of analytics depends on how wisely we integrate new tech, not how quickly we jump on the hype train.