Technical GEO
AI Crawlers and robots.txt: How to Let AI Search Find Your Site
By Akshay Zine, Founder of AuctorisAI · Updated
The short answer
To be cited by AI search, allow the search crawlers in your robots.txt: Googlebot for Google AI Overviews, OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity. Training crawlers such as GPTBot and Google-Extended are separate controls, and blocking them does not remove you from search. Also check that your firewall or bot protection is not blocking these crawlers.
What is the difference between AI search crawlers and training crawlers?
OpenAI documents OAI-SearchBot (used to surface sites in ChatGPT search), GPTBot (used for model training) and ChatGPT-User (user-initiated fetches). Each can be controlled separately in robots.txt. [1] Perplexity documents PerplexityBot for search indexing and Perplexity-User for user-initiated requests. [2] Google-Extended is a token that controls some Gemini uses and does not affect Google Search. [3]
A robots.txt template for AI visibility
If your goal is maximum visibility, allow search and AI crawlers to reach your public pages and block only private areas. Example rules: “User-agent: OAI-SearchBot / Allow: /”, “User-agent: PerplexityBot / Allow: /”, “User-agent: Googlebot / Allow: /”, plus “Disallow: /admin” for private sections and a “Sitemap:” line. Decide separately whether to allow training crawlers such as GPTBot. [4]
- List the crawlers you want to allow and block.
- Keep private paths disallowed for every user agent.
- Add your sitemap URL.
- Check CDN, WAF and bot-protection settings so verified crawlers are not blocked.
How do I check whether AI can read my site?
Fetch your homepage without JavaScript and confirm your key text, title, description and headings are in the HTML. The free AuctorisAI Action Report includes a homepage AEO Readiness check that inspects the actual server-returned HTML, schema and indexing directives.
Common questions
If I block GPTBot, will ChatGPT stop citing me?
Not necessarily. OpenAI documents that OAI-SearchBot, used for ChatGPT search, is controlled separately from GPTBot, its training crawler.
Does robots.txt hide a page from search?
robots.txt controls crawling, not indexing. To keep a page out of Google Search, use a noindex directive on a crawlable page.
Sources and references
Definitions of GEO and AEO reflect common industry usage. Platform-specific claims are attributed below; guidance and product behavior can change.
See what this means for your brand.
Start with the free AI Visibility Action Report, or explore AuctorisAI’s GEO and digital authority services if you need help turning evidence into a plan. No rankings, rich results or AI citations are guaranteed.