Block AI Crawlers: The 15 Bots That Matter

Most lists that claim to help you block AI crawlers are copy-pasted and dangerously wrong about the two tokens that actually matter. Sorting them properly is not an abstract taxonomy exercise. It is the single decision that determines whether your content vanishes from AI answers while training continues — or vice versa.

We maintain the crawler registry in lib/ai-crawlers.ts that powers techpotions’ free AI robots.txt generator. Every agent string and description was verified against the operator’s own crawler documentation. The registry holds 15 verified bots across four categories, and that four-way split is this article’s structure, because the categories map directly to what blocking costs you.

Two tokens almost everyone gets wrong

Before the list, the single most important correction to make, and almost every listicle on this query gets it wrong: Google-Extended and Applebot-Extended are not crawlers. They are robots.txt tokens — product controls that govern whether your content is used for Gemini and Apple foundation-model training.

Blocking Google-Extended does not affect Google Search crawling, Google ranking, or regular Applebot search indexing. People block them believing they are opting out of AI Overviews, and are actually opting out of nothing they think they are, while leaving search indexing completely untouched.

Platforms have started wiring these tokens into one-click controls. Cloudflare’s managed robots feature, released mid-2025, lets you add AI crawler rules through a dashboard toggle rather than editing a raw file — but the underlying token logic above still applies.

Block AI crawlers: the four categories that decide the cost

Every AI crawler we track belongs to one of four categories. The category tells you the cost of blocking it.

• Training crawlers scrape pages to feed a model that may never cite you. Blocking them is a defensive data decision.
• Assistant crawlers fetch pages to answer a live user’s question and can cite and link you. Blocking them removes you from AI answers.
• Search crawlers underpin traditional or AI search indexes. Blocking them is a visibility trade-off with ranking consequences.
• Scrapers operate with little transparency and rarely honor robots.txt. A block is a declaration of intent, not a guarantee of enforcement.

Blocking everything is the lazy answer, and it costs AI-answer visibility. The entire point of this list is to let you decide bot by bot, with your eyes open.

Training crawlers: default-deny if you opt out of training

These bots feed foundation models. They will not send you traffic or citations. The trade-off is simple: if you want to block AI crawlers from training on your work, start here.

Crawler	Operator	What it does	What you give up by blocking it
GPTBot	OpenAI	Crawls content that may be used to train OpenAI’s generative foundation models, including the models powering ChatGPT.	The clearest training opt-out. Blocking this has zero impact on whether ChatGPT can answer a live user query about your page — that is a separate assistant crawler.
CCBot	Common Crawl	Crawls the web to build the Common Crawl dataset, a massive open repository of web pages used as training data by many LLM builders.	You opt out of a foundational training dataset used across the industry, not just a single model. It sends no user traffic.
ClaudeBot	Anthropic	Crawls content that may be used to train Claude. Separate from Anthropic’s assistant crawler.	You lose no AI-answer visibility. Claude’s live-fetch assistant crawler is a completely different agent.
Google-Extended	Google	A token, not a bot. Controls whether content crawled by Google’s regular search crawler can be used for Gemini foundation-model training.	Opting out does not affect ranking, Google Search crawling, indexing, or inclusion in AI Overviews. You are only blocking Gemini training.
Applebot-Extended	Apple	A token, not a bot. Controls whether content crawled by regular Applebot can be used for Apple foundation-model training.	Does not affect Apple search indexing. Blocking this does nothing to Siri or Apple Intelligence live answers.

Assistant crawlers: block these and you leave the conversation

Assistant crawlers fetch your page to answer a real user’s question. They can cite and link you. This is the category where blanket blocking most often hurts traffic, because the site owner thought they were stopping training.

Crawler	Operator	What it does	What you give up by blocking it
ChatGPT-User	OpenAI	Fetches pages on behalf of ChatGPT users who ask questions that require live web retrieval. It cites sources.	You remove your content from ChatGPT’s answers entirely. No citation, no referral traffic, no visibility in the assistant that millions use for real-time research.
OAI-SearchBot	OpenAI	Crawls for OpenAI’s web search product. Used for surface-level search indexing rather than training.	You disappear from OpenAI’s native search experience, distinct from ChatGPT’s browsing. An SEO visibility call, not a training call.
PerplexityBot	Perplexity	Fetches pages to generate Perplexity’s AI answers with inline citations.	Blocking it means Perplexity cannot pull your page into answers, removing a growing source of cited traffic.
Cohere-AI	Cohere	Fetches web content to power Cohere’s Retrieval-Augmented Generation pipeline, with citations.	You drop out of an enterprise RAG stack. If your content answers high-stakes or business queries, this is a concrete loss.

Search crawlers: the visibility trade-off

These bots feed search indexes, including AI search. Blocking them carries ranking consequences — not theoretical ones.

Crawler	Operator	What it does	What you give up by blocking it
Googlebot	Google	Decades-old search crawler powering Google Search, Discover, and now the AI Overviews index.	You leave the Google search ecosystem entirely. There is no separate token to opt out of AI Overviews while keeping search ranking. Googlebot feeds both.
Bingbot	Microsoft	Crawls for Bing Search and powers the retrieval layer behind Microsoft Copilot answers.	You vanish from Bing Search and from Copilot citations. One block, two surfaces.
DuckDuckBot	DuckDuckGo	Crawls for DuckDuckGo’s web index, which also underpins DDG’s AI answer features.	You leave a search engine that, while smaller, fuels AI answers that do cite. Visibility trade-off.

Scrapers: a block is the right signal, but enforcement is uneven

These crawlers aggregate content, often for commercial datasets or internal models. The compliance story is thinner, and four entries in our registry carry explicit caveats. Bytespider is the clearest case: it has no official crawler documentation at all, so our entry states exactly that rather than pretending a policy exists. A block here signals intent. It is not a technical guarantee.

Crawler	Operator	What it does	What you give up by blocking it
Bytespider	ByteDance	Widely observed to crawl for TikTok and Douyin recommendations, potentially feeding internal research datasets. No official docs exist.	There is no documented user-facing benefit to keeping it. The cost is speculative scraping with zero transparency.
FacebookBot	Meta	Crawls for article previews, ad quality, and internal ML research. The boundary is deliberately blurred.	You lose rich Facebook share cards and article previews. The same agent may also feed training, and Meta does not separate them with distinct tokens.
Amazonbot	Amazon	Crawls for Alexa, product search, and AWS AI services. Amazon’s documentation states metadata enrichment as a secondary purpose.	Blocking it can affect how your products appear in Alexa answers and the Amazon shopping index, but the crawl does not offer referral traffic back to your site.

How to block AI crawlers without breaking visibility

The right approach is a per-bot robots.txt rule set that maps to your actual goals. A site that wants training opt-out but AI-answer visibility might look like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

Enter fullscreen mode Exit fullscreen mode

This blocks foundation-model training while keeping your content in ChatGPT and Perplexity answers. We built the AI robots.txt generator to produce these split-category files in seconds, because the mistake we kept seeing was people pasting a blockade that also removed them from AI answers. For the specification behind agent instructions beyond robots.txt, see llms.txt examples.

If you are tuning this as part of a broader AI visibility play, our Growth Labs team works on the question from the traffic side: what you gain by being in the models people are actually asking.

FAQ

Does blocking Google-Extended stop my content appearing in AI Overviews?

No. Blocking Google-Extended tells Google not to use crawled content for Gemini foundation-model training. It does not affect Google Search crawling, ranking, or inclusion in AI Overviews. Blocking it does not opt you out of AI Overviews.

Can blocking AI crawlers hurt my visibility in AI-powered answers?

Yes. Assistant crawlers like ChatGPT-User fetch pages to answer a live user’s question and can cite and link you. If you block the assistant category, those tools cannot retrieve your content at all, and you become invisible in their answers. The typical mistake is a blanket block that removes sites from AI answers while the owner intended only to opt out of training.

Why can’t I just use a NoAI meta tag to stop training?

Because it is not a reliable mechanism against the scrapers that ignore it. Bytespider has no official crawler documentation, so there is no verifiable opt-out procedure. A <meta name="robots" content="noai"> tag is not honored by all training crawlers and offers no protection against bad-faith scraping. A robots.txt rule is the stronger, more widely recognized signal.

Ready to set the exact rules that match your risk tolerance and visibility goals? Start with our crawler registry tool.