772 | Unpacking LLMs.txt with Carolyn Shelby

EDGE of the Web - The Best SEO Podcast for Today's Digital Marketer

Erin welcomes Carolyn Shelby, the Principal SEO at Yoast and a renowned authority in technical and enterprise SEO. Carolyn brings decades of hands-on experience from her pioneering days in digital marketing, working with brands like Disney’s ESPN, Tribune Publishing, and major nonprofits. The conversation kicks off with a surprising twist—Carolyn’s unique title as Queen of the micronation Ladonia—before diving into her role at Yoast and their latest innovation: the LLMs.txt file generator.

Carolyn explains how this new file helps websites communicate their most valuable content directly to large language models like ChatGPT and Google’s AI, streamlining the way future search agents discover and answer questions with information from your site. We explore what inspired Yoast’s push to roll out LLMs.txt to over 13 million sites, what website owners should include in their files, potential industry pushback, the adoption challenge with search giants, and how this moment could change the way websites optimize for AI-driven search results.

Key Segments:

• [00:01:46] Introducing Carolyn Shelby, Senior SEO at Yoast
• [00:03:09] Queen of the Micronation Ladonia?
• [00:07:56] What is the LLMS.txt file?
• [00:08:59] LLMS.txt is a Treasure Map
• [00:14:38] A New File, along with Robots.txt and Sitemap.xml
• [00:15:41]  What inspired Yoast to create this LLM text file?
• [00:17:12]  EDGE of the Web Sponsor: PreWriter.AI
• [00:18:22] LLMS.txt proposed by Jeremy Howard (Sept, 2024)
• [00:22:37] Standard Uniformity and Acceptance?
• [00:24:43] Housekeeping
• [00:29:37] LLM Markdown Effort Questioned: Exploitation?
• [00:31:41] LLMs Lack Memory at Inference
• [00:34:07] EDGE of The Web Sponsor: Inlinks (WAIKAY)
• [00:36:09] Pushback on the LLMS.txt file

Thanks to Our Sponsors!

PreWriter.AI: https://edgeofthewebradio.com/prewriter

Inlinks WAIKAY https://edgeofthewebradio.com/waikay

Follow Our Guest

Twitter: @cshel

LinkedIn: https://www.linkedin.com/in/cshel/

Resources

Learn about Ladonia (DONATE!): https://www.ladonia.org/about/

Carolyn’s Posts on LLMS.txt:

https://www.cshel.com/ai-seo/how-llms-interpret-content-structuring-for-ai-search-unedited-version/

https://searchengineland.com/llms-txt-isnt-robots-txt-its-a-treasure-map-for-ai-456586