Why can't ChatGPT see my website when it shows up in Google?
ChatGPT most often can't see a site that's in Google because it searches other search engines, including Bing. If Bing doesn't know the site…
AI bots, the programs ChatGPT, Perplexity, Claude and Google use to read web pages, get onto your site if nothing stops them along the way. Three things can stop them: the robots.txt file, Cloudflare or server security, and a page that responds too slowly or without text. You can check each one in a few minutes. Server logs show whether the bots really come in.
Below: which AI bots visit websites and what happens when you block one, 4 steps to check, and what to do with the result. Accurate as of October 2026.
AI bots come in three kinds. A search bot collects pages so the assistant can show them in an answer later. A user bot opens a page when someone in a chat asks for a specific URL. A training bot collects text to train models. The kind determines what you lose when you block a bot. Blocking a search bot takes your site out of AI answers. Blocking a training bot doesn’t.
| Company and assistant | Search bot | User bot | Model training |
|---|---|---|---|
| OpenAI (ChatGPT) | OAI-SearchBot | ChatGPT-User | GPTBot |
| Perplexity | PerplexityBot | Perplexity-User | none (PerplexityBot doesn’t collect for training) |
| Anthropic (Claude) | Claude-SearchBot | Claude-User | ClaudeBot |
| Google (AI Overviews, AI Mode, Gemini) | Googlebot | – | Google-Extended (no separate bot) |
| Apple (Siri, Spotlight, Safari) | Applebot | – | Applebot-Extended (no separate bot) |
| Microsoft (Bing, Copilot) | Bingbot | – | – |
Sources: documentation from OpenAI, Perplexity, Anthropic, Google, Apple, as of October 2026. Bingbot is in the table because ChatGPT searches Bing, among others. If Bing doesn’t know a site, ChatGPT won’t find it, even when OpenAI’s bots are allowed in. I wrote about this in the article why ChatGPT can’t see a website that Google can.
If you don't know whether AI bots can reach your site, I'll check it in your logs and server settings.
Book a free callClaudeBot doesn’t decide whether your site shows up in Claude. ClaudeBot collects text for training. Whether Claude finds your site in search depends on Claude-SearchBot and Claude-User. Anthropic says that blocking them may reduce a site’s visibility in answers.
Google-Extended doesn’t turn off AI Overviews. Google says plainly that Google-Extended has no effect on a site’s presence in Search and isn’t a ranking signal. It only limits training and answers in Gemini and some other Google services. AI Overviews and AI Mode use the regular Googlebot. You can switch them off only with nosnippet, max-snippet or noindex, which costs you in regular results too (Google, AI features in Search).
User bots may not follow robots.txt. A visit from ChatGPT-User or Perplexity-User is requested by a human, so these bots treat it like a visit from a browser. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says the rules may not apply to ChatGPT-User.
A robots.txt file is a note on the door, not a lock. An honest bot reads it and obeys, a dishonest one walks right past. For diagnosis, the note is enough, because the search bots of OpenAI, Perplexity, Anthropic, Google and Apple all state that they follow it.
How to check: type yourdomain.com/robots.txt into your browser. The file is made up of groups. Each group starts with a User-agent: line naming the bot, followed by Disallow (stay out) and Allow (come in) rules.
A bot listens only to the group with its own name. It reads the asterisk group (User-agent: *) only when it can’t find its own name (the RFC 9309 standard). That rule creates two traps:
Disallow: / under the asterisk closes the site to every bot without its own group, including AI search bots;Disallow: / in the OAI-SearchBot, PerplexityBot or Claude-SearchBot group takes the site out of search in that assistant, even if the rest of the file opens everything.If you want to block training but keep search, the file can look like this:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Whether to hand your text over for training is the site owner’s call. Once the decision is made, it has to go where the file really comes from. In WordPress, robots.txt is generated by WordPress itself, and an SEO plugin or the host can change it, so check the file in your browser again after every change. OpenAI says its search systems pick up a change after about 24 hours.
A firewall decides before the bot ever reads robots.txt. If it stops the bot at the door, the note on the door doesn’t matter.
Cloudflare. Since July 1, 2026, Cloudflare splits AI bots into three behaviors: Search, Agent (acting on a human’s request) and Training. Each can be blocked across the whole site, blocked only on pages with ads, or allowed. Domains added from September 15, 2026 have Training and Agent blocked by default on pages with ads. Search is allowed, but bots that also collect for training on the side get caught by the training block (Cloudflare, July 2026). The setting is in the domain’s security settings, on the „Configure AI bot traffic policies” screen.
Hosting. Some hosts have their own bot protection. Ask their support whether they block traffic by bot name or by request rate.
Quick test. Send the site a request signed with a bot’s name. On macOS and Linux, type the command below in the terminal. On Windows, type it in PowerShell, but write curl.exe instead of curl, because plain curl runs a different command there.
curl -I -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://yourdomain.com/
A 200 code on the first line of the response means the server lets that name through. A 403 or 503 means something is blocking it. The test has a limit: it only checks blocking by name. A firewall that lets in only the real IP addresses of bots will reject your command and let the real bot in. The opposite happens too: the test passes, but the firewall blocks the real bot by IP address or by request rate. So the logs have the final word (step 4).
A bot that got in needs to get text, fast. The most common problem, content added by JavaScript that the bot can’t see, is covered in cause 4 of the ChatGPT article. The test is „View page source.” Apart from content in the code, three things count: how fast the server responds, how heavy the page is and what sits in the code before the heading.
Server response time (TTFB). TTFB is the time from sending a request to the first byte of the response. OpenAI, Anthropic and Perplexity don’t publish how long their bots wait, so the reference point is Google: according to web.dev, a good time is 0.8 s or less and a poor one is over 1.8 s. You can measure it in PageSpeed Insights or with curl -s -o /dev/null -w "%{time_starttransfer}\n" https://yourdomain.com/ (on Windows: curl.exe and NUL instead of /dev/null).
Page weight. A bot pays for every character it downloads, because text turns into tokens. The HTML page of my article on ChatGPT not seeing a site weighs 55.4 kB, and its Markdown version 11.1 kB (measured on October 14, 2026). Think of a restaurant menu: the same dishes, minus the photos, the borders and the dessert ad. Cloudflare says the same page in Markdown means 80–84% fewer tokens. That’s data about tokens, not about whether AI cites the page more often.
What sits in the code before the heading. Some bots turn the page into text and read it in order, from the top of the code. If the menu sits at the top, the bot reads the menu first and the answer only after that. On webmountain.pl, 38 words of menu stood in the code before the H1 heading. After I changed the order in the code, there are 0, and the page looks exactly the same. Whether this helps with citations is unknown: it’s my hypothesis, not a measured effect.
Server logs record every visit to the site: the IP address, the date, the URL, the response code and the name of the program that asked. Only the logs show whether the real bot came and what it got. You can usually download the access log from your hosting panel.
How to read the log, step by step:
OAI-SearchBot, PerplexityBot, Claude-SearchBot.200 means the bot got the page, 301 a redirect, 403 a block, 404 a missing page, 429 too many requests at once, and 5xx that the server couldn’t cope..md versions, llms.txt.This is what one log line with a bot visit looks like:
203.0.113.25 - - [14/Oct/2026:09:12:44 +0200] "GET /pozycjonowanie/chatgpt-nie-widzi-strony/ HTTP/1.1" 200 55445 "-" "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"
This is a format example. The address 203.0.113.25 comes from a range reserved for documentation and doesn’t belong to any bot.
A bot name in the log proves nothing, because anyone can sign with it. I tested this on my own site. On October 14, 2026, I sent requests to webmountain.pl signed as OAI-SearchBot, GPTBot, PerplexityBot and ClaudeBot. All of them came from my own computer. The server handed each one the page with a 200 code, so for a few seconds I was four AI bots at once in my own logs.
You recognize a real bot by its IP address. Some companies publish lists of their addresses, others tell you to check the address with a reverse DNS lookup. That lookup turns an IP address into a server name. Then you do the reverse: turn the name back into an address and check that you get the same IP. Only a match in both directions confirms anything. How each company does it:
applebot.apple.com, or use the applebot.json list;googlebot.com or google.com domain (Google: verifying Googlebot);search.msn.com. Bing Webmaster Tools also has a „Verify Bingbot” tool.| What you see | What it means | What to do |
|---|---|---|
Disallow: / in a search bot’s group or under the asterisk |
robots.txt shuts the door | remove the block; in ChatGPT the change takes effect after about 24 h |
the curl test with a bot name returns 403 or 503 |
a firewall or Cloudflare blocks by name | allow Search in Cloudflare; ask your host for an exception |
403 or 429 in the logs for IP addresses from the official lists |
a firewall or rate limit stops the real bot | an exception for the IP ranges from the companies’ lists |
the bot gets 200, but the page source has no text |
JavaScript adds the content | server-side rendering (step 3) |
lots of 5xx in the logs or long response times |
the server can’t keep up | page caching, stronger hosting |
| everything is open, but there are no AI bots in the logs | the bot doesn’t know the site exists | Bing index, sitemap, internal links |
When the bot gets in and gets the content, the technical side is fine. From there, the text decides. I collected the points I check in the writing itself in my checklist for writing pages for AI answers.
I went through steps 1–3 on my own site; step 4 is waiting for a log review. Status as of October 14, 2026:
.md version 11.1 kB, plus an /llms.txt file with the site’s table of contents;What happened to the AI writing checklist on a site prepared like this, a week after publication, I describe in the case study: first place in Google and in an AI Overview.
What I don’t know yet: whether real bots fetch the .md versions and llms.txt. The logs will show that. If no bot touches the .md files in 2–3 months, I’ll drop those versions and add a note here.
You can do steps 1–3 yourself in fifteen minutes or so: read robots.txt, check Cloudflare and send a test request. Step 4 takes access to the logs and patience, because the log of an ordinary website runs to thousands of lines a day. Verifying IP addresses and setting firewall exceptions is a job for someone technical.
If you’d like me to check this on your site, from robots.txt and Cloudflare to the logs and tests in ChatGPT, here’s how it works: SEO and AI search visibility.
Author - Michał Wojciechowski
I have run Google Ads accounts myself since 2019 and I create the ads and copy for them. I have run campaigns on 9 markets and worked on projects on 8 more.
I started in graphic design and film, so I look at an ad the way a viewer does. I work directly with business owners: regular calls, decisions based on data, and work on the offer and the landing page, not only on the ads.
One hour on a call is usually enough to see why the sales are not coming in.