Episode #270: AI and the Analyst. We've Got It All Figured Out
The Analytics Power Hour
Release Date: April 29, 2025

Introduction

In Episode #270 of The Analytics Power Hour, hosts Michael Helbling, Tim Wilson, Julie Hoyer, Val Kroll, and Mo delve deep into the evolving relationship between artificial intelligence (AI) and the field of analytics. The conversation navigates the promises and pitfalls of AI, exploring whether it serves as a mere buzzword or a transformative tool capable of reshaping analysts' roles.

The Evolution of AI in Analytics

Michael Helbling kickstarts the discussion by reflecting on AI's journey within the analytics realm. He humorously mentions an AI-generated podcast intro from 2023 that fell short, setting the stage for a candid exploration of AI's advancements and current standing.

Michael Helbling [00:13]: "AI hasn't gone away, and the possibilities, capabilities, and potential of these LLMs are expanding, seemingly by the minute."

Helbling raises critical questions: Is AI just another predictive model, or does it hold the potential to disrupt traditional analytics roles? The hosts agree that while AI has made significant strides, it remains a tool that complements rather than replaces human expertise.

AI's Current Capabilities and Limitations

Julie Hoyer emphasizes that AI isn't yet ready to replace analysts. She points out that while AI can assist with tasks like writing queries, it lacks the nuanced understanding required to derive meaningful business insights.

Julie Hoyer [04:15]: "AI is not ready to just replace us. Even for writing queries, there's no talk to your AI and ask it your business questions and have the data insights come from your data warehouse."

Tim Wilson concurs, highlighting the difference between debugging and AI-generated query writing. He underscores that AI still requires significant human oversight to ensure accuracy and relevance.

Mo introduces an intriguing use case from Canva, where AI assists in generating optimized SQL queries by leveraging the company's top-performing dashboard data. This collaboration showcases AI's potential to streamline data retrieval processes without eliminating the need for human intervention.

Mo [05:35]: "We're not at a point where you don't need a data person involved at all. You still definitely need to QA data."

Real-world Examples and Trials

The hosts discuss various experiments and implementations of AI in analytics. Mo shares Canva's approach to enhancing SQL query generation using AI, which has shown promising results in producing more efficient and accurate queries. However, he notes the necessity of Subject Matter Experts (SMEs) to guide and validate AI outputs.

Mo [06:57]: "I think we might get to a point where we don't need dashboards. Mic drop."

This sparks a heated debate on the future of dashboards. While Mo suggests a day without dashboards might be feasible, Tim and Julie express skepticism, emphasizing the enduring value of visual data representations for business stakeholders.

The Role of Human Expertise

A recurring theme in the episode is the irreplaceable role of human expertise in leveraging AI effectively. Michael Helbling eloquently captures this sentiment:

Michael Helbling [10:04]: "Your source data requires many different other pieces of metadata or parallel data... An inference engine like an LLM can actually come up with something that is not just sort of like that intern level time on site was 42 seconds type of bull crap you get from big agencies."

Val Kroll references Eric Sandersam's concept that while AI excels at problem-solving, it falters in problem definition—a critical aspect where human intuition and business context come into play.

Val Kroll [25:35]: "AI is really good at problem solving and it's getting better and better, but it's not making a lot of progress on the problem defining part of it."

Michael extends this by suggesting that as AI becomes more integrated, it will spotlight individuals who excel in understanding business levers and deriving actionable insights, potentially rendering others obsolete.

The Future of Dashboards and Analytics Tools

The conversation shifts to the potential obsolescence of traditional dashboards. Mo envisions a future where AI-driven tools can provide real-time insights without the need for static dashboards.

Mo [19:48]: "If you can answer that question without having a dashboard, why would you need it?"

However, Tim counters by emphasizing the importance of consistency and the human preference for structured visual data.

Tim Wilson [20:10]: "There is value in consistency. Structure."

The hosts grapple with balancing AI's capabilities with human preferences, acknowledging that while AI can generate visualizations, the tactile familiarity of dashboards remains valuable.

Challenges and Concerns with AI Adoption

A significant portion of the discussion revolves around the challenges posed by AI adoption in analytics. Julie Hoyer shares frustrating experiences where AI misinterpreted business terms, leading to inaccurate analyses that required extensive human correction.

Julie Hoyer [07:10]: "It doesn't know that it's not smart enough. It doesn't know you haven't trained it how to actually... it doesn't have the context around your data."

Tim Wilson voices concerns about the proliferation of AI-generated content leading to information overload and superficial analyses, hindering meaningful decision-making.

Tim Wilson [33:35]: "Most social media promotions that are like AI generated, they're the worst."

Mo introduces the concept of AI acting as a "rubber duck," where the iterative feedback loop between human and machine enhances the quality of work, but also acknowledges the potential for AI to produce subpar outputs without proper guidance.

The Importance of Developing Expertise Amid AI

The hosts passionately discuss the necessity of cultivating expertise in an AI-augmented landscape. They argue that while AI can expedite routine tasks, developing deep analytical skills remains paramount for extracting actionable insights.

Michael Helbling [34:28]: "Knowledge and expertise actually becomes a massive and important filter for how AI is actually going to be beneficial or not beneficial."

Val Kroll and Tim Wilson emphasize that expertise is forged through experience, including making and learning from mistakes—something AI cannot replicate. They caution against over-reliance on AI, which might lead to a workforce that's only "average" without the capacity for innovative thinking.

Tim Wilson [38:52]: "...the nicest thing is AI is nowhere close to doing the analyst job now. Give it two years and my story will change."

Conclusion and Final Thoughts

As the episode winds down, the hosts express a mix of optimism and caution regarding AI's role in analytics. They recognize AI's potential to revolutionize data handling and analysis but stress the indispensable value of human insight and expertise.

Michael Helbling encapsulates the episode's essence by urging analysts to leverage AI's strengths while honing their own skills to navigate the complex landscape.

Michael Helbling [34:09]: "AI is right down the middle in terms of an average... without knowledge, I only could possibly hope for average."

The episode concludes with lighthearted banter and acknowledgments, reinforcing the camaraderie among the hosts and their shared commitment to advancing the analytics community.

Key Takeaways

• AI as a Tool, Not a Replacement: AI enhances analytical tasks but cannot replace the nuanced understanding and problem-defining capabilities of human analysts.

• Human Expertise is Crucial: The effectiveness of AI in analytics heavily depends on the user's expertise to guide, validate, and interpret AI outputs.

• Challenges in AI Adoption: Issues like context misinterpretation, information overload, and the necessity for extensive data preparation hinder seamless AI integration.

• Future of Dashboards: While AI may reduce reliance on traditional dashboards, the need for structured visual data representation remains significant.

• Developing Expertise: As AI takes over routine tasks, the emphasis shifts to cultivating advanced analytical skills to extract meaningful insights and drive business decisions.

Notable Quotes

• Michael Helbling [10:04]: "An inference engine like an LLM can actually come up with something that is not just sort of like that intern level time on site was 42 seconds type of bull crap you get from big agencies."

• Julie Hoyer [07:10]: "It doesn't have the context around your business... People aren't connecting those dots of all the steps in between."

• Tim Wilson [33:35]: "Most social media promotions that are like AI generated, they're the worst."

• Val Kroll [25:35]: "AI is really good at problem solving and it's getting better and better, but it's not making a lot of progress on the problem defining part of it."

For more insights and discussions on analytics, reach out to the hosts on LinkedIn, the Measure Slack chat, or via email at contact@analyticshour.io. Remember to rate and review the podcast to help others discover valuable content. Stay tuned for more episodes of The Analytics Power Hour where experts share their knowledge and experiences to empower your analytical journey.