AI Crawlers and robots.txt: Who Can Use Your Content

You control AI crawlers through robots.txt, one bot name at a time. The important point is that the big AI companies now run separate bots for training, for search and for fetching a page when a user asks, so "block AI" is not one switch. For most business sites the sensible default is to allow the search and user-request bots, which can send you visibility, and decide separately about the training bots.

This matters because blocking everything with a single rule can quietly remove you from the answers people now get from ChatGPT, Claude or Google's AI features, while allowing everything hands your content to model training without a conscious decision. Below is how the main bots divide up, what robots.txt can and cannot do, and a setup we would suggest to a typical small business.

What kinds of AI bots visit your site?

Three jobs, and the good news is that the major companies have started to separate them.

**Training crawlers** collect pages to help build future models. Blocking them keeps your content out of future training runs; it does not affect whether you appear in search results.

**Search crawlers** index pages so an AI product can cite and link to them in answers, much like a classic search engine. These are the ones that can send visitors, so blocking them usually makes you less visible, not more private.

**User-triggered fetchers** load a page because a person asked an assistant about it or pasted a link. They act on one person's request rather than crawling the web on their own, which is why companies treat them differently from the other two.

Which bot is which?

Names change, so treat this as a snapshot and check each company's own documentation before you edit anything. As of November 2026, the main ones are these.

How does Google fit in?

Google works differently from the others, and it is the most commonly misunderstood. Google-Extended is not a separate crawler. It is a token you put in robots.txt to say that Googlebot-collected content should not be used for Gemini training and grounding. Google states that it does not affect inclusion or ranking in Google Search.

Ordinary Googlebot is what feeds Search, and AI Overviews are part of Search. So disallowing Google-Extended does not take you out of AI Overviews. If you want to be visible there, the work is the usual one: clear, fast, well-structured pages, which we cover in our guide to AI Overviews and GEO. Google has also described other, separate controls for how Search uses snippets, so read Search Central before assuming one line does everything.

What can robots.txt actually do, and what can't it?

robots.txt is a polite request defined in a public standard (the Robots Exclusion Protocol, RFC 9309). Well-behaved crawlers read it before fetching and follow it. It is not a lock. Anything you can open in a browser can be fetched by a bot that chooses to ignore the file, and some scrapers do.

Three practical limits follow from that. First, rules take time: OpenAI notes that changes can take around 24 hours to be picked up, so do not test your edit five minutes after saving. Second, rules cover future crawling, not what a company already collected. Third, user-initiated fetches may not follow the same rules: OpenAI says that because ChatGPT-User actions are started by a person, robots.txt may not apply to them, whereas Anthropic says all three of its bots honour robots.txt. Vendors differ, and they revise their documentation, which is another reason to check the source.

If something truly must not be read, robots.txt is the wrong tool. Put it behind a login, or do not publish it. For content you simply would rather not see in training sets, robots.txt is a reasonable and widely respected signal.

What setup do we suggest for a typical business site?

A business site exists to be found, so for most of our readers the default is to stay open to the bots that surface and cite you, and make a deliberate choice about training.

Allow the search and user-request bots from OpenAI and Anthropic (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User). Blocking those can reduce how often you appear when someone asks an assistant about your field, and Anthropic itself warns about that trade-off. Then decide about the training bots: GPTBot, ClaudeBot and the Google-Extended token. Block them if you publish original writing, photography or research that you do not want absorbed into models without credit. Leave them open if you are a small service business and broad awareness of your name is worth more to you than the principle.

There is no universally right answer, and we would be wary of anyone who claims one. A photographer or publisher and a plumber have different interests. What matters is that the choice is deliberate, written down and reviewed once or twice a year.

A minimal robots.txt reflecting the cautious option (open to search, closed to training) looks like this in plain text: a group for GPTBot with Disallow: /, a group for ClaudeBot with Disallow: /, a group for Google-Extended with Disallow: /, and no rules at all for the search and user bots. Each group is a User-agent line followed by a Disallow line. Remember that blocking applies per host, so a subdomain needs its own file.

One detail on old rules: Anthropic has retired the Claude-Web and anthropic-ai names, so lines that mention only those no longer do anything. If your robots.txt was copied from an article written a year or two ago, it is worth a review.

How do you check it worked?

Open yourdomain.com/robots.txt in a browser and read it as the bot would: is every group where you intended, and is nothing blocked by accident, for example a leftover "User-agent: * Disallow: /" from a staging site? Then look at your server logs or hosting analytics for the bot names above. Seeing them confirms the visits happen; seeing a blocked bot stop, a day or so after the change, confirms the block works.

Be careful with log claims on a small site: user agents can be faked, so a name in a log proves very little on its own. Vendors that publish IP ranges let you verify real visits; for most small businesses, trusting the file plus a periodic look at the logs is enough.

Where do we come in?

On sites we build, robots.txt, the sitemap and structured data are generated by code, so changing a policy is a one-line edit rather than a plugin hunt. If you want an opinion on which bots to allow for your kind of business, or want your current file reviewed, start from our web design service or look at the pricing, and tell us what you publish and who you want to find it.