Voices of Search Podcast: “LLM.txt files — Trend or Trash?”

Date: September 12, 2025
Host: Tyson Stockton
Guest: Duane Forrester, Unbound Answers

Episode Overview

This episode dives into the recent buzz around LLM.txt files within the SEO and content marketing space. Tyson Stockton and guest Duane Forrester critically evaluate whether these new text files are a practical trend for managing crawler behavior or simply unnecessary "trash." The discussion draws parallels to past SEO “innovations” and underscores best practices for site owners looking to manage how web crawlers, including those used by AI models, interact with their content.

Key Discussion Points & Insights

1. What are LLM.txt Files?

• The concept of LLM.txt files is compared to earlier SEO inventions like Moz's Domain Authority and Google's now-defunct PageRank visualizations.
• These files are posited as a "new" protocol for Large Language Model (LLM) crawlers, but are not currently recognized or followed by major bots.

[00:53, Duane Forrester]:
"LLM TXT is an invention, kind of like domain authority for Moz. It's their own makeup. The engines don't look at domain authority to make any decisions."

2. Technical Protocols: What Do Crawlers Actually Use?

• Every legitimate crawler follows the long-standing robots.txt standard—nothing new is needed.
• The Creative Commons bot (CCBot), which gathers data for many AI models, already follows robots.txt.

[01:19, Duane Forrester]:
"That's the protocol. All crawlers follow that protocol. Robots TXT... The engines don't look at domain authority to make any decisions. They don't factor that into anything on their end."

• Legacy ranking signals like PageRank were designed for human comprehension, but didn’t influence engines in the ways people believed. Exposing technical scores led to attempts at system manipulation.

[02:15, Duane Forrester]:
"What the PageRank score taught us was that people wanted to game scores like that... even though I don't deserve to be ranking well, I am ranking well. And I'll take the traffic and the revenue, and as long as I don't pop on a radar in Mountain View, I'll be okay."

3. Is there any Use Case for LLM.txt?

• There is currently no evidence that search or AI crawlers recognize or use LLM.txt.
• Adding arbitrary .txt directives serves no practical purpose beyond perhaps confusing site owners or wasting resources.

[03:05, Duane Forrester]:
"Things like the LLM txt, those are made up conceptually... No, you don't [need it], because the new crawlers are still just crawlers and they follow robots.txt."

• If in the future new protocols are introduced, the ecosystem will adapt—but that time is not now.

4. Best Practices for Robots.txt and Sitemaps

• Invest in mastering correct robots.txt syntax: Many webmasters misuse this file, inserting ineffective or incorrect directives.
• Do audit your robots.txt: Inadvertent misconfigurations can block valuable bots (e.g., AI systems like CCBot) from crawling your site.
• Default behavior: Bots will crawl everything by default unless explicitly disallowed. There’s no “do crawl” directive—only “do not crawl.”

[04:08, Duane Forrester]:
"The default for every robot is do crawl. That is the default for a bot. It's its only reason for existence is to crawl a URL. So there is no do crawl directive. There is a do not call directive where you tell it what not to go into."

[04:51, Duane Forrester]:
"...everybody audit your robots.txt. If somebody in your IT department is blocking the bots from these AI systems, oh, buy them a pizza, have a conversation. You need to get that rolled back. They need access."

Notable Quotes & Memorable Moments

• Immediate Reaction to LLM.txt:

  "Trash. Total trash. I cannot get that answer out fast enough. Total trash. Right?"
  — Duane Forrester [00:53]

• Historical Parallel:

  "LLM TXT is an invention, kind of like domain authority for Moz...For years I wanted to create one when I was at Bing...But there was never value in exposing it because what PageRank taught us was that people wanted to game scores like that."
  — Duane Forrester [01:03–02:17]

• Actionable Advice:

  "Spend the time to get to learn how the actual syntax should look like...I see robots.txt files daily with all of this 'go here, go here, go here.' The crawlers ignore all that crap. You're building it, maintaining it, you think you're having some influence...Nothing."
  — Duane Forrester [04:00]

Timestamps for Key Segments

• 00:43 — Introduction to the Topic: LLM.txt (trend or trash?)
• 00:53 — Duane's Immediate Take: "Total trash" verdict
• 01:03–03:05 — Parallels with Domain Authority & PageRank — Invention vs practical utility in SEO
• 03:05–03:53 — What Crawlers Actually Use — Current best practices and why LLM.txt isn't needed
• 04:00–04:51 — How to Properly Use Robots.txt & Site Maps — Avoiding common mistakes, ensuring bot access

Recommendations & Takeaways

• Ignore the Hype: LLM.txt files serve no functional use in the current SEO and AI crawler ecosystem.
• Focus on the Fundamentals: Make sure your robots.txt and sitemap files are correct and maintained.
• Communicate Internally: If bots are being improperly blocked, coordinate with IT to resolve it—“buy them a pizza.”
• Stay Skeptical: Keep an eye on new SEO trends, but scrutinize their legitimacy and practical impact before implementing changes.

This episode is a must-listen for SEO strategists and webmasters seeking clarity amidst rapid industry change—reminding us that fundamental best practices and cautious adoption of new trends are still the most effective path forward.