Podcast Summary: Software Engineering Daily - “Agentic AI at Glean with Eddie Zhou”
Release Date: April 22, 2025
In this insightful episode of Software Engineering Daily, host Sean Falconer engages in an in-depth conversation with Eddie Zhao, a founding engineer at Glean and former Google engineer. The discussion centers around the evolution of Glean from an enterprise search company to a pioneer in agentic AI, exploring the engineering and design considerations essential for building advanced AI-driven productivity tools.
1. Evolution of Glean: From Enterprise Search to Agentic AI
Eddie Zhao begins by outlining Glean’s foundational vision, emphasizing that while enterprise search was the initial focus, the company has consistently aimed to enhance knowledge workers' efficiency across various tasks.
“With enterprise search we were meeting knowledge workers in a little slice of their sort of job to be done... freeing them up to do more things.”
[01:25]
Sean Falconer probes into how Glean transitioned to integrating agentic reasoning systems, to which Zhao explains that agentic AI broadens their assistance both in understanding user needs earlier and in executing tasks post-information retrieval.
“All we've done as we've evolved from Glean search to assistant and now to this agent platform is sort of broaden that segment...”
[01:25]
2. Challenges with Current AI Models and Data Context
The conversation delves into the limitations of large language models (LLMs) in understanding company-specific data. Zhao highlights the crucial need for context injection—integrating an organization's internal data seamlessly into AI operations to enhance relevance and accuracy.
“They’re not getting better at knowing your company's knowledge... figuring out how to implement context injection is really important.”
[04:25]
Falconer emphasizes that even less advanced models can outperform sophisticated ones if they access the right data, underscoring the importance of data liberation and selective information provision.
“You can have essentially a lower power model that has access to the right data... to generate a meaningful response.”
[04:57]
3. Defining a Reasoning Agent
When asked to define a reasoning agent, Zhao presents a flexible framework, acknowledging diverse interpretations while emphasizing agents’ ability to formulate and execute plans using available tools.
“A reasoning agent is something that can, given a set of tools, formulate a plan to satisfy an input and then go and execute those tools.”
[06:25]
He contrasts agents with Retrieval-Augmented Generation (RAG) systems, explaining that agents extend beyond retrieval to execute actions and manage multi-step processes.
“Agents are simply an extension of RAG, where the content being generated may not be the response to the user, it might be the next step in a plan.”
[08:03]
4. Technical Challenges in Building Agents
The discussion moves to the complexities of designing agents, particularly managing unbounded execution and ensuring controlled workflows. Zhao suggests implementing fixed execution limits and leveraging research indicating performance drops with excessively long reasoning tokens.
“You might allow for a fixed number of executions... there's a sharp decrease in performance once the thinking tokens become too long.”
[13:37]
They also explore multi-agent systems as a solution to scalability issues, advocating for a decentralized approach where specialized sub-agents handle distinct tasks.
“Our internal approach is a little bit more, okay, yes, you do have a central agent, but the tools... can delegate more to those other agents.”
[15:08]
5. Managing Identity and Permissions
Zhao explains how Glean leverages its existing identity infrastructure to ensure agents operate within appropriate access boundaries, maintaining security and relevance based on user roles and permissions.
“We can leverage our entire identity infrastructure and platform... certain tools can only execute if you have access to them.”
[24:29]
6. Debugging and Error Handling in Agentic Systems
Addressing the complexity of debugging dynamic agent workflows, Zhao likens it to tracing through a graph of interconnected systems, identifying breakdowns by tracking inputs and outputs at each stage.
“It's composed together... you need to be able to say, okay, at the high level, where can I track this down... and do that trace.”
[25:07]
7. Guardrails Against Hallucinations and Incorrect Information
Falconer raises concerns about AI hallucinations, prompting Zhao to discuss the inherent difficulties in ensuring generated content’s accuracy. Glean adopts a dual approach of offline evaluation and real-time monitoring to mitigate these risks.
“It's a really hard problem... as an ML engineer's perspective of diagnosing, it does matter where the part of the system is breaking down.”
[28:33]
8. Building and Deploying Agents: In-House vs. External Tools
When asked about the development tools for agents, Zhao notes that Glean employs a mix of proprietary systems and open-source frameworks, allowing flexibility while adhering to performance and security standards.
“It's a blend... our principle is to try to reuse where possible.”
[31:06]
9. Internal Use and Dogfooding
Zhao highlights Glean’s commitment to dogfooding, with internal teams actively building and utilizing agents to refine use cases and enhance platform robustness.
“Folks are building internal agents... trying to build those using that same suite of tools from low code to otherwise.”
[37:19]
10. Measuring Success and Use Cases for Agents
On evaluating agent success, Zhao emphasizes usage metrics and long-term engagement as primary indicators, alongside outcome-based measures tailored to specific use cases.
“Usage is always king for any product... it depends on the use case.”
[38:02]
11. Comparing Agents with Simpler Workflows
The conversation addresses when to adopt agentic solutions versus simpler prompt-based workflows. Zhao advocates for a seamless user experience where complexity is abstracted, allowing users to start simple and escalate to agents as needed.
“Ideally they don't need to think about the level of abstraction... push them up the complexity curve as needed.”
[39:08]
12. Key Technical Challenges and Future Directions
Zhao identifies tooling and evaluation suites as significant hurdles, stressing the need for advanced tools to assess and refine agents effectively. He underscores the importance of scalable evaluation methodologies to drive reliability and user trust.
“Tooling and the evaluation suite of tooling... is still sort of a little bit behind.”
[40:09]
Conclusion
The episode concludes with Zhao expressing excitement about the advancements and ongoing challenges in agentic AI. He reiterates Glean’s dedication to building effective, scalable, and secure agent systems that empower knowledge workers.
“We covered so much here... Thanks for some great questions and it was really awesome talking.”
[42:54]
Sean Falconer echoes this sentiment, appreciating the deep dive into the complexities and innovations driving Glean’s agentic AI journey.
“Really enjoyed it and cheers.”
[43:02]
This episode offers a comprehensive exploration of agentic AI within enterprise settings, highlighting both the potential and the intricate challenges of integrating advanced AI systems into everyday workflows. Eddie Zhao’s insights provide valuable guidance for engineers and companies aiming to harness the power of AI to enhance productivity and decision-making.