The Stack Overflow Podcast

Episode: "Search engine bots crawled so AI bots could run"

Release Date: January 6, 2026
Host: Ryan Donovan
Guest: Robert Lester (Akamai Data Scientist)

Overview

This episode dives deep into the explosive evolution of bot traffic on the Internet, particularly focusing on the surge of AI-driven bots from companies like OpenAI, Google, and Anthropic. Host Ryan Donovan and Akamai Data Scientist Robert Lester discuss what these bots are, how they differ from traditional search crawlers, the implications for web infrastructure and businesses, and speculate on the future of automated agent activity online. Instead of a simple "bot or not," the conversation explores evolving strategies for detection, management, and the blurry boundaries between helpful automation and disruptive scraping.

Key Discussion Points & Insights

1. Robert Lester's Background & Perspective

• Lester began his career studying ancient languages and transitioned into computer science/data science, enjoying problem-solving and "data storytelling."
  • [00:47] Lester: "It kind of led me to a natural evolution towards language and logic problems that got me into computer science and engineering...data storytelling and crafting."

2. Defining & Classifying Modern AI Bots

• Traditional bots (e.g., Googlebot) have indexed the web for 15+ years.
• Modern AI bots often blur lines between search, training, and user-driven fetch activity.
• Companies like Google already possess vast, frequently updated indexes; others (OpenAI, Anthropic) fetch data more ad hoc via user triggers.
  • [01:48-03:48]

"We're kind of moving towards more of identification and intent of these bots... moving kind of away from bot or not, because that is seemingly a less important question at this point."
— Robert Lester [04:59]

3. Growth & Distribution of AI Bot Traffic

• AI bot traffic is still a small percentage of all validated bot traffic (~1%), but has grown 400% across industries within a year—an "incredible increase" closely watched by infrastructure providers.
  • [04:06]

"As far as raw numbers, we're still only looking at this stuff as about a percent... but this is a massive growth... up I think 400% across all industries."
— Robert Lester [04:06]

• Traffic is not evenly distributed: commerce (retail, hospitality) is most heavily targeted, reflecting industries that require constant data updates.
  • [11:00]

"If you were to guess what the top industries were going to be targeted by these AI bots... It's actually commerce."
— Robert Lester [11:16]

• AI bots show unpredictable, sometimes "insane" behavior, with request rates fluctuating wildly—especially after new model releases (e.g., ChatGPT updates) [13:12-15:51].

"After their GPT5 release a lot of stuff went just kind of insane... But yeah, it went insane. And then it seemed later that they dialed it back..."
— Robert Lester [13:12]

4. Changing Uses: From Training Data to Agents & Transactions

• Early AI bot activity focused on broad data collection ("scraping for training").
• Increasingly, user-driven agents fetch data on-the-fly or act as intelligent intermediaries ("agents").
• Some agents are starting to perform transactions—like buying products at point of sale—raising new challenges and questions for retailers and customers alike.
  • [15:55]

"We're starting to see agents interacting at point of sale, which is something that we're not entirely sure how the public is going to react to..."
— Robert Lester [15:55]

5. Business Impact and Mitigation Strategies

• Businesses must weigh bot activity: for commerce/retail, bots can enhance visibility; for publishers, they erode click-through and ad revenue.
• Companies are adopting varied, case-by-case mitigation strategies—from licensing to outright blocking.
  • [07:37]

"Mitigation isn't the only number that we're going for though. We have seen a rise in the number of customers that are mitigating bots, these AI bots, and on a case by case basis."
— Robert Lester [07:37]

• Akamai approaches bot management as a nuanced management problem, not merely a threat vector.

6. Bot Detection: Challenges & Techniques

• Akamai uses aggregated analytics, behavioral pattern recognition, and, increasingly, AI-driven models to identify bot traffic.
• Cooperative bots self-identify, but many do not; Akamai must rely on behavioral signals, network telemetry, and continuous model updating.
  • [18:33]

"We do factor in a lot of features, whether it be something like network telemetry, whether we're starting to look at the actual behavior of how these things are working... we're looking at everything."
— Robert Lester [18:40]

7. Rapid Evolution, Risks, and Opportunity

• The bot landscape is still "the wild, wild west," with standards and norms changing rapidly.
• There is both risk (disruption, unpredictability) and significant opportunity (automation, new markets) for those who adapt early.
  • [19:34]

"The first people who are able to game this in their favor are going to be massive winners... this will be an important part of the new online economy and the Internet of things."
— Robert Lester [19:42]

8. Looking Ahead: The Expanding Arena

• The path forward is wide open; companies should prioritize visibility and proactive bot management.
• The holidays bring spikes in bot activity—old threats (like "Grinch bots") are being joined by new, AI-driven forms.
  • [21:18]

"The best message to take away from it all is just how wide open this arena is right now... Being prepared is the best possible step forward. So get in touch with your bots."
— Robert Lester [21:18]

Notable Quotes & Moments

• "[Bot traffic is] not a massive needle mover yet, but the growth rate is what we're more interested in." [04:06] — Robert Lester
• "It's a management problem, not necessarily a threat vector." [07:37] — Robert Lester, on Akamai’s approach to bot traffic
• "It's still the wild, wild west out there." [19:25] — Robert Lester
• "Maybe you need to learn how to sell to agents now, which is a totally different question." [16:38] — Robert Lester
• "The first people who are able to game this in their favor are going to be massive winners." [19:42] — Robert Lester

Timestamps for Key Segments

• 00:47 — Robert Lester's background and transition to tech
• 01:48–03:48 — Classification of modern AI bots vs. traditional crawlers
• 04:06 — AI bot traffic growth statistics
• 07:37 — Mitigation efforts and business impacts
• 11:00 — Industries most targeted by AI bots
• 13:12–15:51 — "Insane" bot behavior after model releases
• 15:55 — AI agents acting at point of sale
• 18:33 — Behavioral detection of stealth bots
• 19:42 — Risks and opportunities in the evolving bot landscape
• 21:18 — The importance of proactive bot management

Conclusion

This episode shines a spotlight on the fast-moving frontier of AI bot activity on the web, unraveling the complexities and huge (sometimes unpredictable) implications for businesses, technologists, and users alike. It’s a world where bots both help and hinder, and where the difference between agent, bot, and user blurs more by the day. As Robert Lester sums up, “being prepared is the best possible step forward. So get in touch with your bots.”