How to check whether AI crawlers can read your website

A marketing manager asks this when AI answers skip the brand, and a developer asks it after a robots.txt change. This guide gives four reads in order. They are the robots.txt rule, the homepage a plain request receives, the server log, and a quote from ChatGPT's own read. Each read proves one fact and nothing more. For the user agent string and the robots.txt line of every crawler, read the AI crawler user agents table. For the free checks around this question, read the free GEO tools guide.

Our project asks the prompt "How do I check whether AI crawlers can read my website?". In the 28 days to 30 October 2026, Grok and ChatGPT ran 62 web searches behind their answers to it. ChatGPT searched for the official robots.txt documentation of GPTBot, ClaudeBot and Google-Extended, 7 times each.

How do you check whether AI crawlers can read your website?

You check whether AI crawlers can read your website in four reads, and each read proves one fact. An AI crawler is a program that an AI company runs to read web pages for training, for search, or for one person's question.

Each platform answers from the pages that its own crawlers read. The table maps each of the 7 platforms FullReach AI measures to the crawlers that feed it, in the catalog read on 30 October 2026.

The free AI crawler checker makes the first two reads on any site in seconds. The user agents guide gives each crawler's user agent string and robots.txt line.

What does the robots.txt check tell you?

The robots.txt check tells you which rule decides each crawler's access, and whether that rule names the crawler. Each crawler has one of three verdicts:

• Allowed: the crawler may request the root and an ordinary page.
• Blocked: the crawler may not request the root.
• Blocked, with exceptions: the crawler may request the root, but the group's default rule refuses other paths. The pair Disallow: / above Allow: /$ reads this way.

A scoped rule such as Disallow: /admin/ leaves the verdict at Allowed. A site with no robots.txt permits every crawler. A robots.txt that cannot be read gives no verdict, never an allowed one.

To read the file by hand:

1. Open /robots.txt on your site in a browser.
2. Search the file for the crawler's token, such as GPTBot.
3. Read the Disallow and Allow lines of that group.

The user agents guide explains which group decides when no group names the token. It also holds the verdicts of the six sites our project monitors.

An Allowed verdict shows that a crawler may read your pages. It does not show whether the answers that the crawler feeds name your brand.

Can AI crawlers read a JavaScript site?

A crawler that runs no JavaScript reads only the text the server sends, so the check is a plain request to the homepage. FullReach AI's checks set a floor of 800 characters of visible text before any script runs. Below that floor, the checks report that the page did not render on the server. On 30 October 2026, the fullreach.ai homepage served 12,576 characters to a plain request. On the same day, the five rival homepages our project monitors served 13,467 to 39,589 characters. One of the five held part of its content in an inline data block, which a crawler may still read.

The same request reads the page directives. A robots meta tag or an X-Robots-Tag header can carry noindex, nosnippet, max-snippet, noarchive or nocache. The directive nosnippet tells Google not to use the page's content as a direct input to its AI features. A data-nosnippet attribute hides one part of the page the same way. On 30 October 2026, one of the five rival homepages marked one such part.

Agent Analytics›Crawlability

Example

Check	Pipedrive
Does the homepage open to a plain request?	YesHTTP 200
Can they read the page text?	Yes4,812 characters
Can ChatGPT open it?	Not confirmed2 tries on Sep 30
Does the page opt out of AI answers?	No

Does the server log prove that a crawler came?

A log row proves a request once you check its address against the operator's published list. The user agents guide gives each operator's address list and how to check a row against it.

FullReach AI gives each logged request one of three labels:

• Verified: the address sits in the operator's published list.
• Spoofed: the operator publishes a list, and the address sits outside it.
• Unverifiable: the operator publishes no list, or the log row carries no address.

To read the log by hand:

1. Search one day of the access log for the crawler's token.
2. Check the address of one row against the operator's published list.
3. Read the status code of the row. A 403 or a 429 is a refusal that robots.txt cannot show.

A firewall or a bot rule can refuse a crawler that robots.txt allows. The log shows that refusal, and robots.txt does not.

Can you prove that ChatGPT reads your homepage?

Yes, with a quote. A request from our own servers that forges ChatGPT's user agent proves nothing, because a firewall can treat the two differently. So the daily audit asks OpenAI's API, with web search, to open the project's homepage and return one sentence from it. The audit sends the first six words of a sentence from the middle of the page. The title, the H1 and the meta description never qualify, because a search snippet shows them. Each try carries a fresh query string, so no cached copy can answer.

A verbatim match proves that ChatGPT's fetcher read the page that day. Two misses leave the read unproven, because a miss proves no cause. On 30 October 2026, the check read the fullreach.ai homepage and returned the sentence in full. The check runs for the project's own site only, and only for ChatGPT.

What do you change when a rule blocks a crawler?

Change the rule that the verdict names, then read the log the next day. The order follows the cost:

1. A blocked search crawler costs citations today. Allow OAI-SearchBot, PerplexityBot, Googlebot and bingbot first.
2. A blocked fetcher for a person's question costs an answer somebody asked for. Allow ChatGPT-User and Perplexity-User.
3. A blocked training crawler costs a place in a future model. Decide GPTBot, ClaudeBot and Google-Extended by your policy.

Our guide on why your brand is missing from AI answers lists blocked crawlers as one of five causes and gives a test for each. Our guide on how to earn a mention in ChatGPT answers reads the cited pages and lists the moves that follow. FullReach AI repeats the robots.txt and homepage reads every day, for the project's site and each monitored rival.

What to do next

1. Open your robots.txt. Find the group that decides for GPTBot, OAI-SearchBot and PerplexityBot.
2. Run the AI crawler checker on your site. Change each blocked verdict by the table above.
3. Search one day of your log for the crawlers. If robots.txt allows a crawler and the log never shows it, check the firewall.

For how FullReach AI collects the answers, read the methodology.