Free tool
robots.txt & llms.txt generator (with AI-crawler rules)
Control Googlebot, GPTBot, ClaudeBot & friends — and give AI search your site's map
robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /private User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: /
llms.txt
# Site Name ## Pages - [Home](https://example.com/) - [Services](https://example.com/services) - [Blog](https://example.com/blog) - [Contact](https://example.com/contact)
Upload both files to your site root. /llms.txt helps ChatGPT/Perplexity cite your pages correctly.
robots.txt controls which crawlers see your site — and in 2026 that includes AI bots like GPTBot and ClaudeBot, not just Google.
llms.txt is the new companion file: a plain-language map of your site that AI assistants read to cite you correctly. Both are free one-time files at your domain root.
Frequently asked questions
Should I block AI crawlers?
Most small sites should allow them — being cited by ChatGPT and Perplexity is free traffic. Block GPTBot only if you don't want your content used for model training (note: OAI-SearchBot controls ChatGPT search separately).
Where do these files go?
Your domain root: yourdomain.com/robots.txt and yourdomain.com/llms.txt. If you're on Vercel/Netlify, drop them in your public/ folder.
Is llms.txt an official standard?
It's an emerging convention (proposed 2024), already adopted by thousands of sites and read by major AI products. Cheap insurance to publish.