Robots.txt Best Practices for AI Crawlers: Allow vs Disallow Rules

In the past, robots.txt was simple: allow Googlebot and Bingbot, block admin folders, and you were done.

Today, there are dozens of specialized AI crawlers traversing the web. Some crawl to index real-time search citations (which you want). Others scrape to build offline foundation models without attribution (which you may wish to control).

Here is how to configure a pristine, modern robots.txt for maximum AEO impact.

1. The Definitive 2026 AEO Robots.txt Template

# Standard Search Engine Crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Generative AI Answer Engines (Search & Citation Bots)
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

# Disallow Private & Administrative Endpoints
User-agent: *
Disallow: /api/private/
Disallow: /admin/
Disallow: /dashboard/

# Sitemap Declaration
Sitemap: https://signalouts.com/sitemap.xml

2. Common Pitfalls to Avoid

• The Accidental Wildcard Disallow: Placing User-agent: * Disallow: / at the top of your file will instantly de-index you across all AI engines.
• Missing `ChatGPT-User` Directive: When a user enters a link directly into ChatGPT, OpenAI uses the ChatGPT-User user-agent. If blocked, ChatGPT will tell the user it cannot access your site.