Podcast Summary: AI + a16z – "Giving New Life to Unstructured Data with LLMs and Agents"

Date: June 6, 2025
Host: Guido Appenzeller (a16z)
Guest: Anant Bhardwaj (Instabase Founder & CEO)
Special Notes: Co-Host Derek introduces and concludes; transcript skips ads/disclosures.

Episode Overview

This episode explores the evolution and future of automating unstructured data using large language models (LLMs) and AI agents, focusing on enterprise applications. Instabase founder Anant Bhardwaj recounts his research journey (from MIT to Silicon Valley), provides a technical history of approaches to unstructured data, and discusses real-world use cases transforming business processes. The conversation addresses technical breakthroughs, challenges with AI adoption (including reliability and compliance), and a vision for federated, agent-driven automation in the enterprise.

Key Discussion Points and Insights

1. Defining and Understanding Unstructured Data

• Unstructured Data Defined:
  Anything that can't be neatly stored in SQL tables—PDFs, images, scans, emails, mixed documents (02:15).

  "Anything that cannot be put into nice database tables where you can run SQL, anything that is not that is unstructured data."
  — Anant Bhardwaj, 02:15

• Enterprise Challenge:
  Businesses have critical processes relying on unstructured information (immigration forms, loan packets, insurance paperwork).

2. Early Approaches: The Pre-LLM Era

• Manual and Brittle Solutions:
  • Templates: Hard-coded positions for information extraction, break easily (04:41).
  • Rule-based: Searching for key phrases, prone to error.
  • Hand-crafted ML models: Labor-intensive, limited by document variation.
  • Program synthesis: Generating code (e.g., regex) to automate extraction, worked for limited cases.
• Quote:

  "The techniques at that time were very rudimentary… As soon as you scan differently things will break."
  — Anant Bhardwaj, 04:41

3. Enter Transformers and LLMs

• Technical Breakthroughs:

  • Early transformer models (BERT) struggled with document layouts.
  • Instabase innovates by adding positional XY encoding to document tokens, leading to significant gains in layout understanding (06:30–07:57).
  • "Instalm": Their BERT-like model with 2D attention.
  • Industry trends: Now a standard technique for 2D document data.

• LLMs like GPT-3/ChatGPT:

  • ChatGPT’s release (2022) changed the game by handling text-based documents with strong accuracy, but with caveats.
  • Lesson: "Size matters" — bigger models often perform better, but have practical and reliability limitations (08:18).

4. The Modern Enterprise Solution: Automation and Reliability

• How Instabase Approaches the Problem:

  • Layered, auditable workflow: Splitting packets, table detection, cross-validation, schema extraction, automated pipelines (09:40–13:45).
  • Critical requirement: Reliability and explainability.

    "LLMs is not all you need because …if it goes beyond the context window, then that's a problem…How do you know you didn't miss anything?"
    — Anant Bhardwaj, 09:40

• Error Handling and Human-in-the-loop:

  • Recognizing that neither humans nor AI are infallible.
  • Systems flag uncertainties or exceptions to human operators.

    "You have to build systems around [AI]... it's going to be a lot of investment."
    — Anant Bhardwaj, 14:16

• Customer Use Case:

  • Indian bank offering loans over WhatsApp: Conversational, document-uploaded in chat, real-time decisioning (17:46).

    “I've never seen lending being done conversationally over WhatsApp. This is insane...the customer experience is fundamentally very different.”
    — Anant Bhardwaj, 17:53

5. Reliability, Predictability, and Enterprise Compliance

• Shift in Acceptance Criteria:

  • Enterprises (e.g., banks) accept nonzero error rates—provided errors are predictable and auditable (15:21).

    “They don't care about 99% accuracy. You can be 90% accurate or even 80%... just tell us which 20% need to be reviewed.”
    — Anant Bhardwaj, 15:21

• Barriers to Adoption:

  • Slow-moving compliance, legal, and audit processes.
  • Key enterprise concerns: Data security, auditability, and ability to explain AI decisions (21:07–22:56).

6. The Future: Agents and Federated Automation

• What are Agents?

  • Often misused term—sometimes just fancy prompt chains. True agents act with some autonomy, make decisions, and can interact with other agents or systems (22:56–23:36).

• Build-time vs. Runtime Autonomy:

  • Agentic workflows are promising for drafting solutions, but determinism is needed at runtime for auditability.

    “The agents…are not guaranteed to deterministically always go in one path. In general people don't like runtime inconsistencies…runtime has to be consistent.”
    — Anant Bhardwaj, 23:36

  • Best practice: Use AI/agents to generate the initial solution ("compile time"), human review/approval, then run deterministically in production.

    “I do not believe that autonomous agent would be a runtime phenomena. However, there would be a build time or compile time phenomena…”
    — Anant Bhardwaj, 24:21

• Vision: Federated AI / Decentralized Automation

  • Future of automation is AI-powered, federated, and decentralized— many agents acting on behalf of an organization, dynamically discovering and communicating (27:12–29:11).

    "AI will drive automation in a significant way. RPA would be fully eaten by AI automation and the future is likely going to be more of decentralized federated execution."
    — Anant Bhardwaj, 29:12 & 29:33

7. Technical Advances and Evolving Products

• Integration with Enterprise Systems:
  • Moving beyond just extracting structured data, future systems operate existing enterprise applications directly using AI agents.
  • Model Context Protocol (MCP) allows dynamic tool discovery and action, but brings new problems: Authentication, error handling (32:09).
• RPA vs. AI Automation:
  • RPA (Robotic Process Automation) is brittle with unstructured data; AI automation aims to replace RPA by handling the full workflow (31:50–33:51).

Notable Quotes & Memorable Moments

• [04:41] “The techniques at that time were very rudimentary. Templates…as soon as you scan differently, things will break.”
  — Anant Bhardwaj

• [07:47] “That produced great results because the attention is now not just looking at the sequence of tokens, but also XY coordinate in the two dimensional space, which is really, really cool from the perspective of document layout understanding.”
  — Anant Bhardwaj

• [08:18] “Lesson held: size matters.”
  — Guido Appenzeller

• [09:40] “LLMs is not all you need … the right way to solve this is how do you know how to split this particular packet into a bunch of things we care about.…You need a complex workflow under the hood that is explainable, that is auditable, that is guaranteed to be accurate and correct.”
  — Anant Bhardwaj

• [15:21] “More important is predictability. I think people are fine with errors as long as errors are predictable. When errors are not predictable, that's where the problem is.”
  — Anant Bhardwaj

• [17:53] “I've never seen lending being done conversationally over WhatsApp. This is insane.…the customer experience is fundamentally very different.”
  — Anant Bhardwaj

• [23:36] “If you just give agents the same goal and same set of tools…they are not guaranteed to deterministically always go in one path. In general, people don’t like runtime inconsistencies.…runtime has to be consistent.”
  — Anant Bhardwaj

• [29:33] “The bet that we are taking is that AI will drive automation in a significant way. RPA would be fully eaten by AI automation and the future is likely going to be more of decentralized federated execution.”
  — Anant Bhardwaj

• [35:01] “It allows you to do things much, much faster. And the third one is fundamentally changes customer experience in a very significant way. So I think there are all the business reasons for enterprises to adopt these things now. It's just about how to make this work.”
  — Anant Bhardwaj

Timestamps for Major Segments

• 02:15 – Defining unstructured data; MIT research
• 04:41 – Early automation attempts and their limits
• 06:30–07:57 – Innovations in document layout LM modeling
• 08:18 – The impact of OpenAI's ChatGPT and model scale
• 09:40–14:16 – How Instabase handles unstructured data workflows and ensures reliability
• 15:21 – Enterprise expectations: Predictability over perfection
• 17:46–19:50 – Conversational, AI-powered lending on WhatsApp; user experience transformation
• 21:07–22:56 – Main enterprise barriers: auditability, data security, compliance
• 22:56–24:21 – Discussion of “agents” in workflows; build time vs. runtime use
• 27:12–29:33 – Future: federated, decentralized agent frameworks
• 29:49–33:51 – Technical advances, RPA replacement, model context protocol, and the importance of compile time controls

Conclusion

This episode provides a comprehensive look at the technological evolution and enterprise adoption of AI for unstructured data. Bhardwaj emphasizes that blending advanced LLM techniques with engineered workflow systems, human oversight, and emerging agent-driven automation can fundamentally transform enterprise operations. He articulates a shift from brittle, rule-based automation to federated, AI-powered workflows that are reliable, explainable, and customer-centric—signaling a future where routine enterprise labor is increasingly handled by intelligent, collaborative digital agents.