How do I check whether AI bots can access my website?

AI bots, the programs ChatGPT, Perplexity, Claude and Google use to read web pages, get onto your site if nothing stops them along the way. Three things can stop them: the robots.txt file, Cloudflare or server security, and a page that responds too slowly or without text. You can check each one in a few minutes. Server logs show whether the bots really come in.

Below: which AI bots visit websites and what happens when you block one, 4 steps to check, and what to do with the result. Accurate as of October 2026.

In this article (11)

Which AI bots visit websites and what they’re for

AI bots come in three kinds. A search bot collects pages so the assistant can show them in an answer later. A user bot opens a page when someone in a chat asks for a specific URL. A training bot collects text to train models. The kind determines what you lose when you block a bot. Blocking a search bot takes your site out of AI answers. Blocking a training bot doesn’t.

Company and assistant Search bot User bot Model training
OpenAI (ChatGPT) OAI-SearchBot ChatGPT-User GPTBot
Perplexity PerplexityBot Perplexity-User none (PerplexityBot doesn’t collect for training)
Anthropic (Claude) Claude-SearchBot Claude-User ClaudeBot
Google (AI Overviews, AI Mode, Gemini) Googlebot – Google-Extended (no separate bot)
Apple (Siri, Spotlight, Safari) Applebot – Applebot-Extended (no separate bot)
Microsoft (Bing, Copilot) Bingbot – –

Sources: documentation from OpenAI, Perplexity, Anthropic, Google, Apple, as of October 2026. Bingbot is in the table because ChatGPT searches Bing, among others. If Bing doesn’t know a site, ChatGPT won’t find it, even when OpenAI’s bots are allowed in. I wrote about this in the article why ChatGPT can’t see a website that Google can.

Hi, I'm Michał.

If you don't know whether AI bots can reach your site, I'll check it in your logs and server settings.

Book a free call

Three AI bot names that mislead people most often

ClaudeBot doesn’t decide whether your site shows up in Claude. ClaudeBot collects text for training. Whether Claude finds your site in search depends on Claude-SearchBot and Claude-User. Anthropic says that blocking them may reduce a site’s visibility in answers.

Google-Extended doesn’t turn off AI Overviews. Google says plainly that Google-Extended has no effect on a site’s presence in Search and isn’t a ranking signal. It only limits training and answers in Gemini and some other Google services. AI Overviews and AI Mode use the regular Googlebot. You can switch them off only with nosnippet, max-snippet or noindex, which costs you in regular results too (Google, AI features in Search).

User bots may not follow robots.txt. A visit from ChatGPT-User or Perplexity-User is requested by a human, so these bots treat it like a visit from a browser. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says the rules may not apply to ChatGPT-User.

Step 1: what your robots.txt file says about AI bots

A robots.txt file is a note on the door, not a lock. An honest bot reads it and obeys, a dishonest one walks right past. For diagnosis, the note is enough, because the search bots of OpenAI, Perplexity, Anthropic, Google and Apple all state that they follow it.

How to check: type yourdomain.com/robots.txt into your browser. The file is made up of groups. Each group starts with a User-agent: line naming the bot, followed by Disallow (stay out) and Allow (come in) rules.

A bot listens only to the group with its own name. It reads the asterisk group (User-agent: *) only when it can’t find its own name (the RFC 9309 standard). That rule creates two traps:

  • Disallow: / under the asterisk closes the site to every bot without its own group, including AI search bots;
  • Disallow: / in the OAI-SearchBot, PerplexityBot or Claude-SearchBot group takes the site out of search in that assistant, even if the rest of the file opens everything.

If you want to block training but keep search, the file can look like this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Whether to hand your text over for training is the site owner’s call. Once the decision is made, it has to go where the file really comes from. In WordPress, robots.txt is generated by WordPress itself, and an SEO plugin or the host can change it, so check the file in your browser again after every change. OpenAI says its search systems pick up a change after about 24 hours.

Step 2: is Cloudflare or a firewall stopping AI bots

A firewall decides before the bot ever reads robots.txt. If it stops the bot at the door, the note on the door doesn’t matter.

Cloudflare. Since July 1, 2026, Cloudflare splits AI bots into three behaviors: Search, Agent (acting on a human’s request) and Training. Each can be blocked across the whole site, blocked only on pages with ads, or allowed. Domains added from September 15, 2026 have Training and Agent blocked by default on pages with ads. Search is allowed, but bots that also collect for training on the side get caught by the training block (Cloudflare, July 2026). The setting is in the domain’s security settings, on the „Configure AI bot traffic policies” screen.

Hosting. Some hosts have their own bot protection. Ask their support whether they block traffic by bot name or by request rate.

Quick test. Send the site a request signed with a bot’s name. On macOS and Linux, type the command below in the terminal. On Windows, type it in PowerShell, but write curl.exe instead of curl, because plain curl runs a different command there.

curl -I -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://yourdomain.com/

A 200 code on the first line of the response means the server lets that name through. A 403 or 503 means something is blocking it. The test has a limit: it only checks blocking by name. A firewall that lets in only the real IP addresses of bots will reject your command and let the real bot in. The opposite happens too: the test passes, but the firewall blocks the real bot by IP address or by request rate. So the logs have the final word (step 4).

Step 3: what an AI bot gets once it’s in

A bot that got in needs to get text, fast. The most common problem, content added by JavaScript that the bot can’t see, is covered in cause 4 of the ChatGPT article. The test is „View page source.” Apart from content in the code, three things count: how fast the server responds, how heavy the page is and what sits in the code before the heading.

Server response time (TTFB). TTFB is the time from sending a request to the first byte of the response. OpenAI, Anthropic and Perplexity don’t publish how long their bots wait, so the reference point is Google: according to web.dev, a good time is 0.8 s or less and a poor one is over 1.8 s. You can measure it in PageSpeed Insights or with curl -s -o /dev/null -w "%{time_starttransfer}\n" https://yourdomain.com/ (on Windows: curl.exe and NUL instead of /dev/null).

Page weight. A bot pays for every character it downloads, because text turns into tokens. The HTML page of my article on ChatGPT not seeing a site weighs 55.4 kB, and its Markdown version 11.1 kB (measured on October 14, 2026). Think of a restaurant menu: the same dishes, minus the photos, the borders and the dessert ad. Cloudflare says the same page in Markdown means 80–84% fewer tokens. That’s data about tokens, not about whether AI cites the page more often.

What sits in the code before the heading. Some bots turn the page into text and read it in order, from the top of the code. If the menu sits at the top, the bot reads the menu first and the answer only after that. On webmountain.pl, 38 words of menu stood in the code before the H1 heading. After I changed the order in the code, there are 0, and the page looks exactly the same. Whether this helps with citations is unknown: it’s my hypothesis, not a measured effect.

Step 4: server logs show whether an AI bot really visits

Server logs record every visit to the site: the IP address, the date, the URL, the response code and the name of the program that asked. Only the logs show whether the real bot came and what it got. You can usually download the access log from your hosting panel.

How to read the log, step by step:

  1. Download the access log for the last 7–30 days.
  2. Search the file for the bot names from the table above, for example OAI-SearchBot, PerplexityBot, Claude-SearchBot.
  3. Check the response code for each visit: 200 means the bot got the page, 301 a redirect, 403 a block, 404 a missing page, 429 too many requests at once, and 5xx that the server couldn’t cope.
  4. Check whether the IP address belongs to the company whose name the bot used (next section).
  5. See which URLs the bot fetches: articles, the home page, robots.txt, .md versions, llms.txt.

This is what one log line with a bot visit looks like:

203.0.113.25 - - [14/Oct/2026:09:12:44 +0200] "GET /pozycjonowanie/chatgpt-nie-widzi-strony/ HTTP/1.1" 200 55445 "-" "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"

This is a format example. The address 203.0.113.25 comes from a range reserved for documentation and doesn’t belong to any bot.

How to tell a real AI bot from an impostor

A bot name in the log proves nothing, because anyone can sign with it. I tested this on my own site. On October 14, 2026, I sent requests to webmountain.pl signed as OAI-SearchBot, GPTBot, PerplexityBot and ClaudeBot. All of them came from my own computer. The server handed each one the page with a 200 code, so for a few seconds I was four AI bots at once in my own logs.

You recognize a real bot by its IP address. Some companies publish lists of their addresses, others tell you to check the address with a reverse DNS lookup. That lookup turns an IP address into a server name. Then you do the reverse: turn the name back into an address and check that you get the same IP. Only a match in both directions confirms anything. How each company does it:

What to do when something blocks AI bots: result and fix

What you see What it means What to do
Disallow: / in a search bot’s group or under the asterisk robots.txt shuts the door remove the block; in ChatGPT the change takes effect after about 24 h
the curl test with a bot name returns 403 or 503 a firewall or Cloudflare blocks by name allow Search in Cloudflare; ask your host for an exception
403 or 429 in the logs for IP addresses from the official lists a firewall or rate limit stops the real bot an exception for the IP ranges from the companies’ lists
the bot gets 200, but the page source has no text JavaScript adds the content server-side rendering (step 3)
lots of 5xx in the logs or long response times the server can’t keep up page caching, stronger hosting
everything is open, but there are no AI bots in the logs the bot doesn’t know the site exists Bing index, sitemap, internal links

When the bot gets in and gets the content, the technical side is fine. From there, the text decides. I collected the points I check in the writing itself in my checklist for writing pages for AI answers.

What the AI bot diagnosis looks like on webmountain.pl

I went through steps 1–3 on my own site; step 4 is waiting for a log review. Status as of October 14, 2026:

  • robots.txt doesn’t name any AI bot, so every bot follows the asterisk group. That group closes only the WordPress admin panel and the internal search;
  • test with a bot name: OAI-SearchBot, GPTBot, PerplexityBot and ClaudeBot got the page with a 200 code, 3 requests each;
  • response time (TTFB): from 0.19 to 0.31 s across those 12 requests. I measured from my computer in Poland, and bots connect from their own servers, so their times may differ;
  • weight: the HTML article 55.4 kB, the .md version 11.1 kB, plus an /llms.txt file with the site’s table of contents;
  • code before the heading: 0 words before the H1 (38 before).

What happened to the AI writing checklist on a site prepared like this, a week after publication, I describe in the case study: first place in Google and in an AI Overview.

What I don’t know yet: whether real bots fetch the .md versions and llms.txt. The logs will show that. If no bot touches the .md files in 2–3 months, I’ll drop those versions and add a note here.

When it’s worth asking someone for help

You can do steps 1–3 yourself in fifteen minutes or so: read robots.txt, check Cloudflare and send a test request. Step 4 takes access to the logs and patience, because the log of an ordinary website runs to thousands of lines a day. Verifying IP addresses and setting firewall exceptions is a job for someone technical.

If you’d like me to check this on your site, from robots.txt and Cloudflare to the logs and tests in ChatGPT, here’s how it works: SEO and AI search visibility.

FAQ

Frequently asked questions

Does blocking ClaudeBot remove my site from Claude?
No. ClaudeBot collects text to train Anthropic's models. Whether Claude finds your site in search depends on Claude-SearchBot and Claude-User. Blocking ClaudeBot in robots.txt only means that future text from your site won't be used for training.
Does Google-Extended turn off AI Overviews?
No. Google says Google-Extended has no effect on presence in Search. AI Overviews use the regular Googlebot, so you can limit them only with nosnippet, max-snippet or noindex, which also affect regular results.
Does Perplexity follow robots.txt?
PerplexityBot, the search bot, follows robots.txt. Perplexity-User, which opens a page at a user's request, generally ignores robots.txt according to Perplexity, because a human requested the visit.
How do I check whether GPTBot in my logs is really OpenAI?
Compare the IP address of the visit with the address list OpenAI publishes for that bot (gptbot.json, searchbot.json, chatgpt-user.json). The name in the log alone proves nothing, because anyone can sign with it.
Does my site need a Markdown version for AI bots?
Nobody knows yet. A Markdown version weighs several times less, so a bot uses fewer tokens on it, but there's no public data showing that AI cites it more often. I'm testing this on webmountain.pl in the server logs.

Author - Michał Wojciechowski

Who writes this?

I have run Google Ads accounts myself since 2019 and I create the ads and copy for them. I have run campaigns on 9 markets and worked on projects on 8 more.

I started in graphic design and film, so I look at an ad the way a viewer does. I work directly with business owners: regular calls, decisions based on data, and work on the offer and the landing page, not only on the ads.

Michał Wojciechowski

Campaign not working?

One hour on a call is usually enough to see why the sales are not coming in.