Quick answer
AI companies run separate crawlers for training, search indexing and live user-triggered fetches, and each obeys robots.txt by its own user-agent name. Allowing search and user agents such as OAI-SearchBot, ChatGPT-User and PerplexityBot preserves citation visibility, while training agents such as GPTBot and CCBot can be blocked independently.
What are the key takeaways?
- —Training, search and live-fetch agents are separate: blocking one has no effect on the others.
- —Blocking search and fetch agents removes you from AI answers entirely, which is usually not what the publisher intended.
- —Rules are matched by exact user-agent name, so name each agent explicitly rather than relying on a wildcard group.
- —robots.txt is the only file with permission semantics; llms.txt and meta tags do not grant or deny crawler access.
- —Review the file annually — new agents appear regularly and inherit only your wildcard rule until you name them.
Why are there so many AI user agents?
Each AI company separates its crawling by purpose so that publishers can make different decisions about each. OpenAI runs GPTBot for training corpus collection, OAI-SearchBot for building the index behind ChatGPT search, and ChatGPT-User for fetching a page live because a user's prompt requires it. Anthropic runs ClaudeBot and Claude-User along similar lines. Perplexity runs PerplexityBot for indexing and a user agent for live session fetches. Google separates Googlebot from Google-Extended, which governs generative AI use rather than indexing.
The separation is genuinely useful and almost universally misused. A site that wanted to opt out of training and pasted a broad block into robots.txt has, in most cases, also opted out of being cited in AI answers — a very different decision that nobody explicitly made.
The rule to internalise: training agents affect whether your content shapes a model. Search and fetch agents affect whether your site is named as a source today. They are independent choices and should be made independently.
Which agents should you allow?
If visibility in AI answers is the goal, allow every search and live-fetch agent: OAI-SearchBot and ChatGPT-User, PerplexityBot and its user agent, ClaudeBot and Claude-User, Google-Extended, Bingbot, Applebot and Applebot-Extended where relevant, and the various smaller research and answer crawlers as you encounter them in logs.
Training agents — GPTBot, CCBot, Meta-ExternalAgent and similar — are a separate call. Blocking them costs you nothing in immediate citation terms. Allowing them may, over long horizons, help a model associate your terminology and expertise with your brand. There is no consensus answer, and either choice is defensible provided it was chosen rather than inherited.
Whatever you decide, write it explicitly and comment the file. A robots.txt with a line explaining why each group exists is the difference between a policy and an accident, and it saves the next person from undoing your reasoning.
How do you write the rules correctly?
Robots.txt groups are matched by the most specific user-agent name that applies. Write one group per agent name rather than assuming a wildcard covers it, because an agent that finds a named group for itself ignores the wildcard group entirely — which means a permissive wildcard plus a restrictive named group can produce results opposite to what a skim of the file suggests.
Keep the file simple and flat. Long files with dozens of overlapping path rules are where mistakes hide. Disallow the paths that genuinely should not be crawled — internal search results, session URLs, staging paths, API endpoints — and leave everything else open. Never disallow the CSS and JavaScript needed to render the page, a legacy mistake that still appears in inherited files.
Finish with an absolute sitemap URL. It costs one line and is the cheapest discovery signal available. Then verify by fetching your own robots.txt over HTTPS on the canonical host and confirming it returns 200 with the exact text you expect, since a redirecting or 404ing robots file is treated very differently from a permissive one.
What robots.txt cannot do
It cannot enforce anything. Compliance is voluntary, and while the major AI companies publicly honour it, scrapers that do not identify themselves obviously will not. If content must not be taken, the control is authentication, not a text file.
It cannot remove content that has already been crawled. Adding a disallow today does not retract yesterday's copy from an index or a training set. Withdrawal processes, where they exist, are handled through each company's separate channels.
And it cannot express nuance. There is no way in robots.txt to say 'quote me but do not train on me' beyond the coarse agent-level split the companies themselves provide. Work within that split rather than inventing directives, since unrecognised lines are simply ignored.
How do you audit what is actually happening?
Read your server logs. They are the only ground truth about which agents visit, how often, which URLs they request and what status codes they receive. A crawler receiving 404s or redirects on the pages you care about is a problem no on-page work will fix.
Compare log activity against your intentions. If you allowed PerplexityBot but see no hits in ninety days, the issue is discoverability rather than permission — check the sitemap, internal links and whether the pages are reachable without JavaScript. If you blocked a training agent but still see it fetching, verify the user-agent name is spelled exactly as documented, since a typo silently disables the rule.
Then close the loop with an on-page audit that reads the file the way an agent does. Our readiness audit fetches your robots.txt, checks it against every AI agent name we track, and reports which are permitted, which are blocked and which fall through to a wildcard you may not have intended.
What do people ask most about this topic?
If I block GPTBot, am I removed from ChatGPT answers?
No. GPTBot governs training data collection. ChatGPT's answers cite pages retrieved by OAI-SearchBot and fetched live by ChatGPT-User. Block GPTBot and allow the other two, and your citation visibility is unaffected.
Does a wildcard rule cover every AI crawler?
Only agents that find no group naming them specifically. If a named group exists for an agent anywhere in the file, that agent follows it and ignores the wildcard entirely — a common cause of rules behaving opposite to expectation.
Is blocking AI crawlers good for SEO?
It has no positive effect on classical rankings and removes you from AI answer surfaces. Google-Extended in particular is separate from Googlebot, so blocking it withdraws you from generative experiences without changing how you rank.
How often should I review robots.txt?
At least annually, and whenever you see an unfamiliar agent in your logs. New AI crawlers launch regularly and inherit only your wildcard rule until you decide about them explicitly.
Should I use noai or noimageai meta tags?
They are not honoured by the major AI crawlers and carry no enforcement. Use documented robots.txt user-agent rules for permission decisions, and authentication for anything that genuinely must not be fetched.
Written and reviewed by Jim Vernon, Editor, AI Intelligence International. Last reviewed 2026-08-24. Published by AI Answer Engine and checked against our editorial standards.