AI crawler user agents, with robots.txt examples

A developer asks this question with a robots.txt file open, and a marketing manager asks it after a blocked verdict. This guide is the reference table: each crawler's token, its owner, what it feeds, its user agent string, its address list and its robots.txt line. It ends with a copy-ready robots.txt for a brand that wants AI answers to cite it. For the four reads that prove whether a crawler can read your site, read how to check whether AI crawlers can read your website. For the free checks around the question, read the free GEO tools guide.

In the 28 days to 30 October 2026, ChatGPT ran 27 web searches for an AI crawler's official documentation before it answered our prompts. Grok ran 36 searches that named AI crawlers or robots.txt in the same window.

Which crawlers belong on an AI crawler user agents list?

An AI crawler user agents list names each program that an AI company sends to read web pages, with the token that robots.txt matches. The token is the operator's own spelling, such as GPTBot, and it appears inside the full user agent string. FullReach AI's catalog, read on 30 October 2026, holds 31 crawlers from 14 operators. The table below holds 11 of them, from five operators. They are the crawlers behind the 7 AI platforms our project measures, plus Anthropic's three, which ChatGPT's searches name. The user agent strings come from the operators' pages, read on 30 October 2026. Google's page says the Chrome version inside its string changes over time.

The other 20 crawlers in the catalog feed no platform our project measures. Our guide on the four reads maps each of the 7 platforms to the crawlers that feed it.

What does each crawler feed?

Each crawler feeds one of three things: a training set, a search index, or one person's question. The operator's page, read on 30 October 2026, states which, and the cost of a block follows from it.

• Training: the crawler collects pages for a future model. OpenAI's page says a Disallow for GPTBot signals that the site's content should not train its models. Anthropic's page says a restriction on ClaudeBot signals that the site's future material should stay out of training. Google's page says Google-Extended governs whether crawled content may train future Gemini models. It also governs grounding, which is the supply of indexed pages to Gemini at answer time.
• Search: the crawler builds the index that an AI answer cites. OpenAI's page says a site that opts out of OAI-SearchBot leaves ChatGPT's search answers, but it may still appear as a navigational link. Perplexity's page says PerplexityBot collects nothing for model training.
• A person's question: the fetcher opens one page because somebody asked. OpenAI's page says robots.txt rules may not apply to ChatGPT-User, because a person started the request. Perplexity's page says Perplexity-User generally ignores robots.txt for the same reason. Anthropic's page says a block on Claude-User stops Claude from retrieving the site's content for a user's question.

In the catalog, read on 30 October 2026, four tokens carry an operator's statement that robots.txt may not stop them. Two are in the table above. The other two belong to Meta and Amazon. Google's page, read on 30 October 2026, says Google-Extended does not affect a site's place in Google Search and is not a ranking signal. Our guide on what Google says about AI Overviews and AI Mode summarizes the rest of Google's page on AI features.

How do you prove a request by its address list?

A request proves its operator only when its address sits in the list that the operator publishes. In the catalog, read on 30 October 2026, 17 of the 31 crawlers have such a list. The other 14 have none, so a request that carries their token proves nothing.

The lists differ in shape:

1. OpenAI publishes one JSON file per crawler. Perplexity does the same.
2. Google publishes one file for all its common crawlers. A match proves Google, and the user agent string then names the crawler.
3. Anthropic publishes one file for its three bots. A match proves Anthropic, not which bot came.
4. Microsoft's file for bingbot, read on 30 October 2026, holds 28 IPv4 prefixes and no IPv6 prefix. Its creation time reads 3 January 2024.

Perplexity's page, read on 30 October 2026, says the ranges change and names the JSON files as the source of truth. Anthropic's page, read on 30 October 2026, says an address block may fail as an opt-out, because its bots then cannot read robots.txt. So a firewall rule that allows a crawler should read the current file, not a copy. Our guide on the four reads explains the labels verified, spoofed and unverifiable that FullReach AI gives each logged request.

Which robots.txt line allows or blocks each crawler?

A robots.txt group starts with one or more User-agent lines and holds the Allow and Disallow lines that rule those tokens. A crawler reads the group that names its token. FullReach AI's checker matches the token without regard to case, so gptbot and GPTBot name the same crawler. When no group names a crawler, the checker applies the User-agent: * group.

Three facts from the operators' pages, read on 30 October 2026, change how you write the file:

1. OpenAI's page says each of its settings is independent. A rule for GPTBot says nothing about OAI-SearchBot. The page also says one crawl may serve both uses on a site that allows both.
2. OpenAI's page says its systems take about 24 hours to reflect a robots.txt change. Perplexity's page says up to 24 hours.
3. Anthropic's page says its bots honor the non-standard Crawl-delay line.

FullReach AI's audit of 30 October 2026 read the robots.txt of six sites, ours and five rivals'. Every site allowed all 31 crawlers in the catalog. So robots.txt alone does not tell you whether AI answers name your brand.

What does a robots.txt for a brand that wants to be found look like?

It allows every search crawler and every fetcher for a person's question, and it states a decision on the training crawlers. The file below does that. Replace the sitemap address and the folders you keep private.

# Search crawlers and fetchers: allow, so AI answers can read and cite the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Training crawlers: Allow to appear in future models, Disallow to stay out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /

# Everybody else, Googlebot and bingbot included
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

The training group is the one policy decision in the file. A brand that wants a place in future models keeps Allow: /. A brand that wants its pages out of training changes that one line to Disallow: /. The search group stays as it is in both cases. The two fetchers in the first group may ignore the file by their operators' own statements. Their lines still document the intent.

How do you keep the list current?

Read the operator's page, never a copy of it, because the strings and the address lists change. The table below names the page behind each row of this guide and the day we read it.

FullReach AI reads the robots.txt of your site and of each monitored rival every day against its catalog of 31 crawlers. A blocked crawler is one of the five causes in our guide on why your brand does not show up in AI answers. That guide gives a test for each cause.

What to do next

1. Copy the robots.txt above. Decide the training group's one line, and keep the search group as it is.
2. Run the AI crawler checker on your site. It names the line that decides each crawler in the catalog.
3. Allow each crawler's address list in your firewall or CDN, from the JSON file in the table, not from a copy.

For how FullReach AI collects the answers, read the methodology.