Practical AI Podcast — Episode Summary

Episode: We’ve all done RAG, now what?
Date: September 29, 2025
Host: Daniel Whitenack (B), CEO at Prediction Guard
Guest: Rajeev Shah (C), Chief Evangelist at Contextual AI
Theme: Moving beyond first-wave retrieval-augmented generation (RAG) deployments — real-world lessons, the evolution of AI workflows, and where value is created in applied AI today.

Main Theme & Purpose

This episode explores the shift from the initial excitement and quick demos with Retrieval Augmented Generation (RAG) systems—used widely for knowledge search, customer support, and internal documentation—to the deeper challenges and “next steps” organizations encounter as they bring these systems toward scalable, maintainable real-world impact. The discussion delves into the importance of context engineering, critical misconceptions around model training versus retrieval, evaluation challenges, the evolving roles in AI/data, and navigating hype to achieve business value.

Key Discussion Points & Insights

1. The Arc of RAG: From Pilots to Production

• Initial Hype and Use Cases:

  • RAG became the go-to for making language models useful with organization-specific or sensitive data (e.g., HR docs, healthcare guidelines).
  • [04:46] Rajeev: “RAG today is kind of one of the most important, or it’s one of the most widely used use cases… every company is probably running some type of RAG at this point for searching its internal knowledge, right?”

• Misconceptions About Model Training:

  • Many users confuse the need for LLM fine-tuning with what retrieval accomplishes.
  • [06:11] Daniel: “It almost seems like OpenAI has a separate model for every person on the planet, which is not feasible... there’s kind of this jargon of training thrown around a lot, which is confusing.”
  • Rajeev: Emphasizes that context engineering—providing the right information at the right time—is often much more practical than retraining.

• Scaling Pitfalls:

  • Moving from toy demos to real-world deployment exposes challenges in scaling, latency, accuracy, user interaction diversity, and mounting pipeline complexity.
  • [24:05] Rajeev: “The trouble that people get into... is scaling it up. It’s great on a hundred documents, but now all of a sudden I have to go to 100,000 or a million... Or the accuracy is not what I was looking for... There’s all these kind of trade-offs as you get to production.”

2. Beyond Retrieval: Reasoning, Agents, and Context

• Reasoning in LLMs:

  • The field has shifted from models simply recalling facts to exhibiting “reasoning” (structured, stepwise problem-solving, e.g., multi-step math).
  • [13:32] Rajeev: “They’re doing lots of extra steps and they’re doing these steps in a logical way to better solve a problem... We’ve literally given examples of, hey, this is how I solved this word problem… I want you to learn how to go through these problems step by step.”

• Context Engineering:

  • Effective systems now require orchestration—retrieving, memory management, re-ranking, query reformulation, summarization.
  • [07:07] Rajeev: “What we see inside of AI engineering [is] context engineering... managing interactions with these models. Whether it’s RAG, memory... summarizing conversations...”

• Agents and Autonomy:

  • As tooling matures, line blurs between rigid workflows and more autonomous “agent” systems.
  • [28:33] Rajeev: “We all like the idea of this agent, right? Like something I can give a problem to and it solves a problem... The big trade-offs that developers have today is how much structure, how much babysitting am I doing for this agent?”

3. Achieving Business Value: Hype vs. Reality

• The Science Experiment Trap:

  • Many projects become “science experiments” that either don’t deliver measurable value or fail to integrate with real workflows.
  • [19:23] Rajeev: “I see often what I call science experiments where teams like the latest technology, they go out and run this stuff, but there’s no way for them to actually get that implemented inside the company in a useful way.”

• Organizational Adoption is Harder than the Technology:

  • Embedding even simple AI into workflows brings the hardest challenges (training, change management, integration).
  • [20:39] Daniel: “Part of the hard problem is cracking what actually does provide value to your organization, what can be adopted, how you communicate that, how you tell that story…”

• The 95% Failure Rate (Myth & Reality):

  • While the “95% of AI pilots fail” statistic is often cited, it’s not unique to AI (applies to experimentation in general) and isn’t necessarily a bad thing.
  • [19:23] Rajeev: “You want things to fail… because if something works, you have to maintain it... there’s a cost for something that actually succeeds.”

• Start Simple, Stay Close to Users:

  • Instead of chasing complexity, prioritize simple AI that addresses genuine use cases and deeply involves end users from the start.
  • [22:19] Rajeev: “...once you cut through that, sometimes you figure out that really they don’t necessarily need a fancy GPT-5 model to solve their problems... spending time talking to those end users is going to give you the biggest bang for your buck..."

4. The Evolution of Roles: Data Science, AI Engineering, and Citizen Developers

• Changing Landscape:

  • More business domain experts now build sophisticated solutions themselves.
  • [32:06] Daniel: “I see that middle zone shrinking... domain experts on the business side are actually able to use very sophisticated tools now to kind of self-serve…”

• Data Science Still Needed, but Shifting:

  • The essence of data science—connecting technical and business perspectives, analytics, evaluating impact—remains crucial.
  • [32:55] Rajeev: “At the end of the day, a journalist is a storyteller telling you... For me, the data science is a similar piece... you still need a flexible mind as a data scientist to talk to stakeholders, figure out the coding, the algorithms…”

• Evaluation Remains a Key Gap:

  • Software engineers moving into AI often need to learn how to properly evaluate and monitor models, a strength of traditional data science training.
  • [34:41] Rajeev: “One of the biggest problems they have is with evaluations. And for data scientists, they’re trained on how to do evaluations…”

5. Looking Forward: Blending Old and New AI Tools

• Generative AI Isn’t Always the Best Solution:

  • Many enterprise problems can be solved without LLMs; traditional techniques often work better or more efficiently.
  • [36:14] Rajeev: “There’s a lot of problems inside an enterprise that can be solved without large language models... my worry is the people coming into kind of AI and data science nowadays aren’t seeing those types of problems…”

• Continuous Learning and Experimentation Recommended:

  • Stay up to date but don’t get distracted by hype—combine daily, incremental learning with focus on durable value.
  • [41:40] Rajeev: “Continual learning is the future... I have my own content that I put out at Registics... Newsletters are a nice way to be able to take in all the information that's coming in, but in a little bit of a slower kind of meditative way.”

Notable Quotes & Memorable Moments

• On the Experience of Scaling RAG ([24:05]):
  Rajeev: "The trouble that people get into... is scaling it up. It's great on a hundred documents, but now all of a sudden, I have to go to 100,000 or a million documents. How am I going to do that?"

• On Separating Value from Hype ([17:19]):
  Rajeev: “When you’re in an organization, you really have to think about the problems that you have... it can be very easy to be kind of seduced by the technologies, by what a shiny demo is…”

• On Organizational AI Failure ([19:23]): Rajeev: "You can't expect every initiative, every experiment, everything that you start to succeed, you want things to fail... there's a cost for something that actually succeeds."

• On the Role of Data Scientists ([32:55]): Rajeev: “You still need a flexible mind as a data scientist... where you need a lot of this kind of left brain, right brain stuff. And so it’s still a fairly unique role.”

• On the Balance Between GenAI and Classic Solutions ([40:21]): Rajeev: "There's a great wake of tools that are out there that I still like to kind of point people to — it might not get the most attention, but... a lot of times [it's] a more efficient way of solving your problem as well."

Timestamps for Key Segments

• 00:48–04:46 — Rajeev returns: State of Midwest AI, recap of RAG’s rise, how retrieval fits into practical AI
• 06:11–08:31 — Misconceptions about LLM training, context engineering explained
• 09:59–13:32 — Real-world knowledge, dealing with conflicting “facts,” what counts as “reasoning” for LLMs
• 17:19–22:56 — RAG and the science experiment trap, organizational integration, why 95% of pilots fail
• 24:05–27:25 — Pitfalls of scaling RAG, complexity of pipeline maintenance, the future role of reasoning models in troubleshooting
• 28:33–30:17 — What’s an “agent”? Autonomy vs. plumbing; are we repeating the AI-vs-ML-vs-DS debate?
• 32:06–36:14 — Changing roles: Shrinking middle layer, evaluating models, balancing software and data science mindsets
• 39:04–40:21 — Will LLMs recommend older tools? The persistent need for human orchestration
• 41:40–42:18 — Rajeev’s advice: Newsletters, continual learning, and mixing “old” tricks with new tech

Further Resources

• Rajeev Shah: TikTok, LinkedIn, and his “Registics” content stream for practical AI insights
• Practical AI Podcast: practicalai.fm
• Midwest AI Summit: midwestaIsummit.com
• Relevant newsletters: As recommended by Rajeev for “meditative” AI learning

TL;DR

While everyone can now build a RAG chatbot, the real challenge (and opportunity) for enterprises is in scaling, integrating, and maintaining AI solutions that truly align with organizational workflows. The conversation debunks myths about model “training,” covers the new roles of context and reasoning, and urges listeners to focus on solving concrete problems—often with simple tools—rather than chasing shiny, unproven tech. Data science and context engineering are evolving, not disappearing, and success is still driven by user-centric design, continuous learning, and pragmatic adoption of both old and new AI techniques.