GPTBot and AI crawlers
Updated 2026-09-15
GPTBot is OpenAI's crawler, and it sits alongside a growing set of AI user agents — search-time fetchers, training crawlers and assistant browsers — each with a different purpose. Robots.txt rules decide which of them may read a site, and those rules have direct, verifiable consequences for visibility.
Which agents to decide about
The useful distinction is training versus answering. Training crawlers feed future model memory; search-time fetchers retrieve pages to answer a question being asked right now. Blocking the second removes you from grounded answers where you would otherwise be cited — a direct visibility cost — while blocking the first is a content-licensing decision with slower effects. Because agents are added and renamed over time, the robots file needs review rather than a one-time setup.
Illustrative policy (example configuration)
Allow search-time fetchers so the site can be retrieved and cited in live answers; decide training crawlers as a business question; keep the rules listed explicitly with a comment explaining which agent does what, so the next person editing the file knows what each line costs.
Common mistakes
- Blocking every AI user agent by reflex, then asking why the brand is absent from grounded answers.
- Assuming one rule covers a vendor — search-time and training agents are separate user agents.
- Never revisiting the file. The agent list changes and a stale policy silently becomes a different policy.
Frequently asked questions
- Does blocking training crawlers hurt AI visibility?
- Less directly than blocking search-time fetchers: training affects what a model remembers later, while retrieval decides whether you can be cited in an answer today.
- How can we tell whether AI crawlers actually visit?
- Server logs. User agents are self-declared, but the request pattern for a page after publishing is the cheapest evidence available that a fetcher is reading it.
