Technical
robots.txt and AI bots: what to allow?
Short answer
To be AI-visible, explicitly allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent. Blocking these user-agents bans any citation.
How to do it
- List AI user-agents to allow in robots.txt.
- Test each rule with curl -A 'GPTBot' your URL.
- Keep Disallow only for private paths.
- Reference sitemap.xml at the end of the file.
- Verify robots.txt from Bing and Google search consoles.
Voir la version française
robots.txt et bots IA : que faut-il autoriser ?
Pour être visible dans l'IA, autorisez explicitement GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider et meta-externalagent. Bloquer ces user-agents revient à interdire toute citation.