Should You Block AI Crawlers? A robots.txt Guide for 2026
GPTBot, ClaudeBot, PerplexityBot and Google-Extended explained: which AI crawlers to allow, which to block, and how to stay visible in AI search answers.
Working through a similar problem? Tell us what you want to build.
Legacies is an independent digital product brand based in Bucharest, Romania, founded by Horia Stan and Alexandru Tălnaci, and lately almost every website conversation we have includes the same question: should we block AI crawlers?
For most businesses that sell a service, the answer is no, at least not all of them. Allow the search and user-triggered bots that can send you customers, decide separately about training bots, and protect expensive pages with rate limits rather than a blanket ban. Blocking everything or allowing everything are both guesses if you do not know who is visiting.
Quick answer
- AI bots come in three kinds. Training crawlers collect data for future models, search crawlers build the index AI answers cite, and user-triggered fetchers visit a page when someone asks an assistant a question.
- Block training if you want, keep search. You can disallow GPTBot or ClaudeBot and still allow OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- Google-Extended does not remove you from AI Overviews. Google's AI features use the normal Search index. Blocking Google-Extended only affects Gemini training and grounding.
- Measure before you decide. Check your logs or CDN dashboard to see which bots visit, which pages they read and whether any referrals follow.
Your website now has a second audience
For most of the web's history, a business website had two readers: people and search engines. Create a clear page, make it fast, help Google understand it, turn visitors into customers.
AI changed that. Assistants read pages to answer questions, to train models and to compare providers on behalf of a user. Sometimes that brings a customer who arrives ready to talk. Sometimes it just uses bandwidth without anyone coming back.
Cloudflare measured the imbalance. In its July 2026 post on Attribution Business Insights, it reported AI crawlers with crawl-to-referral ratios from 118:1 up to nearly 50,000:1. That means some bots read tens of thousands of pages for every visitor they send back.
The AI crawlers you should know
Each major AI company now runs separate bots for separate jobs. That is good news, because it lets you make a precise choice. These are the user agents that matter most, based on each company's own documentation:
| Company | Training | Search index | User-triggered |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None declared | PerplexityBot | Perplexity-User |
| Google-Extended (a token, not a crawler) | Googlebot | Not separated |
A few details people get wrong:
- User-triggered fetchers may ignore robots.txt. OpenAI says that because ChatGPT-User acts on a user's request, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt for the same reason. If you need to stop them, you need firewall rules, not a text file.
- OAI-SearchBot decides ChatGPT search visibility. OpenAI says to manage search opt-outs with OAI-SearchBot, not ChatGPT-User. Changes can take about 24 hours to apply.
- Google-Extended is not a separate bot. Google's documentation says it controls whether content can be used for future Gemini models and grounding, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".
What about AI Overviews and AI Mode?
Google's guide to AI features and your website is clear. There are no extra requirements to appear in AI Overviews or AI Mode. A page needs to be indexed and eligible to show a snippet. If you want to limit what appears, the controls are the normal ones: nosnippet, data-nosnippet, max-snippet or noindex. Blocking Google-Extended will not take you out of AI Overviews.
A sensible robots.txt for a service business
robots.txt is a public request, standardized in RFC 9309. Well-behaved bots follow it. It is not a security control. With that in mind, this is a reasonable starting point for a business that wants AI search visibility but prefers not to feed model training. The first three blocks allow AI search bots, the next three opt out of training, and the last one covers everyone else:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /admin/
If you want your content in future models, maybe because you want assistants to know your brand without a live search, allow the training bots too. There is no universally right answer. It depends on whether your content is the product (a publisher, a course, a paid library) or the marketing for the product (a service business, a studio, a local shop). Most of our clients are in the second group, where being read is usually the point.
CDNs now make the choice for you, unless you check
Cloudflare sits in front of a large share of business sites, and its defaults changed this year. In July 2026 it split AI bots into three categories: Training, Search and Agent. From September 15, 2026, new domains onboarding to Cloudflare block Training and Agent bots on pages that display ads by default, while Search bots stay allowed.
If your site is behind Cloudflare or a similar CDN, open the bot settings and check what is actually blocked. We have seen sites where the robots.txt says "welcome" and the firewall says "go away". The firewall wins. We touch on the trade-offs of running logic at the CDN layer in edge computing for small sites.
What to do this month
Start with visibility, not panic:
- Read your logsCheck server or CDN logs for the user agents above. Find out whether AI traffic is significant and which pages it hits.
- Sort your contentA public pricing page and a paid resource library do not need the same policy. Decide what you want read and what you want protected.
- Protect expensive routesSearch endpoints, generated files and dynamic pages may need rate limits or caching no matter who is asking.
- Fix your factsIf an assistant summarizes your business, outdated services and contradictory contact details now cost you more. Make sure every page agrees.
- Review every quarterBots, defaults and referral numbers change. Treat your policy as a setting to revisit, not a permanent stance.
Clarity helps people and machines
The answer is not to make every page read like a database record. A site should stay useful, distinctive and easy to scan. But clarity helps both audiences. A service page that says what you offer, who it is for, where you work, roughly what it costs and how to get in touch is better for a human visitor and less ambiguous for any system trying to summarize it.
That is why we avoid hiding essential information behind animation, images with text baked in or a chain of unclear buttons. Visual character matters. So does a structure that works on a slow connection, with a screen reader or through an AI assistant. Clear headings, real HTML text, accurate structured data and fast pages are the same basics that help in Google.
You do not need an llms.txt file to appear in Google's AI features. Google says you don't need new machine readable files or AI text files. The llms.txt proposal is harmless if you want to try it, but it is not where we would spend the first hour.
If you are not sure how your site looks to these systems, start with our free website audit. It checks speed, mobile, basic SEO and accessibility in a few seconds, no signup. When the issues go deeper, our web optimization service starts at 699 lei (about 130 EUR) and covers technical SEO, Core Web Vitals, schema markup and meta tags. Details are on our services page.
When to handle it yourself and when to bring someone in
Handle it yourself if you run a simple site, can edit robots.txt and just want to choose between training and search bots. The example above and a quick look at your CDN settings are enough.
Bring someone in if your logs show heavy bot traffic, your hosting bill went up, important pages are not being cited anywhere, or your site relies on JavaScript so heavily that bots see an empty page. Those are technical problems with business consequences. The same goes if your business information is spread across a site, a booking tool and a CRM that disagree, a problem we also see when AI agents start reading company data. And if your site has an AI chatbot, check the EU AI Act transparency rules as well.
Frequently Asked Questions
Should I block AI crawlers on my business website?
For most service businesses, no, at least not the search ones. Bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot help assistants cite and link your site. You can still block training crawlers such as GPTBot and ClaudeBot if you do not want your content used for future models, without losing AI search visibility.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses separate bots. GPTBot collects content for model training, while OAI-SearchBot builds the index behind ChatGPT search. You can disallow GPTBot and allow OAI-SearchBot, and OpenAI says robots.txt changes take about 24 hours to apply.
Does blocking Google-Extended stop my site from appearing in AI Overviews?
No. Google-Extended only controls whether your content is used for Gemini model training and grounding. AI Overviews and AI Mode use the normal Google Search index, so to limit how your content appears there you need snippet controls such as nosnippet, data-nosnippet or max-snippet.
Do AI crawlers respect robots.txt?
The declared training and search crawlers from OpenAI, Anthropic, Perplexity and Google say they follow robots.txt. User-triggered fetchers like ChatGPT-User and Perplexity-User may not, because they act on a specific user request. robots.txt is a public request, not a lock, so use firewall or CDN rules if you need to enforce a block.
Do I need an llms.txt file for AI search?
Not for Google. Google states that you do not need new machine readable files or AI text files to appear in AI Overviews or AI Mode. An llms.txt file will not hurt, but clear pages, real HTML text, accurate structured data and good technical SEO matter far more.
How do I see which AI bots visit my website?
Check your server access logs or your CDN's bot analytics for user agents such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and ChatGPT-User. Cloudflare now groups AI bots into Training, Search and Agent categories, which makes it easier to see what each one is doing and decide what to allow.