Podcast Summary: AI + a16z

Episode: Enabling Agents and Battling Bots on an AI-Centric Web

Date: June 13, 2025
Host: a16z (Joel de la Garza, Infra Partner)
Guest: David Mittin (CEO, arcjet)

Overview

This episode explores how the rise of AI agents is transforming web traffic, challenging traditional notions of bots, users, and site security. Joel de la Garza and David Mittin discuss the limitations of old bot-blocking approaches, the nuances of enabling beneficial AI agents, and the emerging need for granular, context-driven access controls. They dive into evolving technical solutions, the complexity of identity and proof-of-humanness, and the likely future where AI agents become the Internet’s primary consumers, requiring a total rethinking of how websites handle automated traffic.

Key Discussion Points & Insights

1. The Shift from Traditional Bots to AI Agents

• Old World vs. New Needs:

  • DDoS remains but is now “almost handled as a commodity” ([02:14] David Mittin).
  • Traditional tools used blunt instruments—blocking IPs or user agents, risking lost legitimate users and revenue.

• Rise of "Agent Experience":

  • Site security built for humans often blocks useful AI agents acting on users’ behalf (e.g., making reservations).
  • The future: Treat agents as “first-class users,” designing for agent experience ([00:31] Host).

2. Nuanced Threat Detection and Allowance

• Blocking is Too Blunt:

  • “Just blocking anything that is called AI is too blunt of an instrument. You need much more nuance.” ([05:08] David Mittin)
  • Modern context: Some bots drive signups and conversions; others may be malicious.

• Application Context is Critical:

  • “You need to know where in the application the traffic is coming to. You need to know who the user is, the session…” ([03:48] David Mittin)

• Quote:

  "If you're running an E-commerce operation...the worst thing you can do is block a transaction..."
  [03:48] – David Mittin

3. Evolving Standards and Controls

• Robots.txt: Still Relevant but Limited

  • Voluntary, widely ignored by “bad” bots, sometimes even used for negative purposes ([06:48] David Mittin).

• Need for Enforceable, Granular Rules:

  • Good bots (Googlebot, OpenAI) usually follow robots.txt; malicious ones don’t, putting the burden on site owners.

4. Taxonomy of AI Agents and Crawlers

[08:06-10:31]

• Different Types of AI Web Traffic:

  • Model training crawlers (often blocked by default)
  • Real-time search/indexing bots (valuable for visibility)
  • User-requested summarization/fetching (acts like a reference checker)
  • Fully headless or browser-based agents (can act autonomously, e.g., for bookings)

• Discernment Is Key:

  • “Blocking all of OpenAI’s crawlers is probably a very bad idea.”
    [10:23] – Joel de la Garza

5. Increasing Sophistication in Detection

• Layered Approach:

  • Start with robots.txt, add IP reputation, analyze user-agent, then fingerprint requests using techniques like JA3/JA4 ([14:30], [15:04]).

• Verification Methods:

  • Reverse DNS, signature fingerprints, Apple’s Privacy Pass, Cloudflare’s new cryptographic signatures ([16:04]).

• Identity Layer:

  • These methods build up “almost like an authentication layer” at every network layer ([15:55] Joel de la Garza).

• Quote:

  "Throughout the whole stack...the idea is you have this consistent fingerprint that you can then apply these rules to."
  [16:04] – David Mittin

6. Agent-Centric Web: Future Implications

• Bots/Agents as Primary Internet Users:

  • Already “50% of traffic is bots” and increasing ([18:15] David Mittin).
  • “We're moving to a world where...agent type activity...will become the primary consumer of everything on the Internet.”
    [17:43] – Joel de la Garza

• Need for Granular Control:

  • Old methods assume malice; the future requires discerning good agents from bad based on context and intent.

7. Proof of Humanness: Still Unsolved

• Identity Proofing:

  • Massive, longstanding challenge (NIST, gig economy, etc.)
  • Digital signatures are “the pure solution,” but poor usability ([21:42] David Mittin).

• AI & ML in Proofing:

  • Classic ML has long been used for detecting bots.
  • LLMs could help analyze patterns and detect non-human activity, but inference cost and speed are current constraints ([23:15]).

• Quote:

  "AI has been used in analyzing traffic for at least over a decade. It was called machine learning."
  [21:42] – David Mittin

8. The Edge AI & Instant Decisions

• Inference Cost and Latency:

  • Speed is becoming viable for microsecond-level decisions.
  • Emerging “super fast inference on the edge” will filter traffic or emails with high accuracy ([24:18]).

• Real-World Applications:

  • E.g., fraud, click spam prevention, instant ad targeting ([25:16]).

• Quote:

  “For advertisers, stopping click spam...and being able to come to that decision before it even goes through your ad model...”
  [25:16] – David Mittin

Notable Quotes & Memorable Moments

• On the problem with legacy solutions:

  "The downside of that is that you probably blocked a lot of legitimate traffic along with illegitimate traffic."
  [03:16] – Joel de la Garza

• On the future of internet traffic:

  "Then we're going to see an explosion in the traffic that's coming from these tools and just blocking them just because they're AI is the wrong answer."
  [18:15] – David Mittin

• On the complexity of agent control:

  "They’re almost like avatars, right? They’re running around on someone’s behalf and you need to figure out who that someone is and what the objectives are and control them very granularly."
  [18:46] – Joel de la Garza

Timestamps for Key Segments

• [00:00 – 03:16] — Introduction, context, and challenges of bots vs. agents
• [03:16 – 06:34] — Shortcomings of legacy blocking, rise of AI-driven traffic benefits
• [06:34 – 08:26] — Robots.txt, voluntary standards, and evolving controls
• [08:26 – 11:52] — OpenAI bot taxonomy, website owner dilemmas
• [11:52 – 16:04] — Technical detection: IPs, user agents, fingerprints, identity layers
• [16:04 – 18:57] — Agent-based Internet usage and future projections
• [18:57 – 21:42] — Proving humanness in a mostly-agent web
• [21:42 – 25:38] — AI/machine learning for identity and fraud detection, edge inference

Tone

The conversation is direct, expert, and slightly irreverent—mixing technical depth with practical, real-world analogies. Both speakers are optimistic yet realistic about the challenges involved in making AI-driven web automation trustworthy, manageable, and beneficial.

Conclusion

A shift to an AI-agent-centric internet requires abandoning blunt, network-level bot-blocking for nuanced, context-rich controls that can distinguish between good and bad automated traffic. The next phase will leverage fast, local AI inference and improved identity signaling to create a more agent-friendly (yet secure and manageable) web.

Recommended For:
Developers, security professionals, product managers, and anyone interested in the intersection of AI, web infrastructure, and the future of online interaction.