AI SEARCH CRAWLER
REFERENCE.

VERIFYING THAT A REQUEST IS GENUINE

Published endpoints · reverse DNS · the layer in front of the origin

A user-agent string is a claim, not an identity. Anyone can send one. What a provider publishes to settle the question is an IP list, a reverse-DNS pattern or a tool — and every endpoint below answered on the date in its row.

The table scrolls sideways.

Two things the documentation adds

Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. Anthropic · DOCUMENTED · 2026-09-05

The IP list is published for recognising Anthropic’s crawlers. Anthropic states that blocking those addresses is not a reliable way to opt out — robots.txt is the documented control.

If you’re using a Web Application Firewall (WAF) to protect your site, you may need to explicitly whitelist Perplexity’s bots to ensure they can access your content. Perplexity · DOCUMENTED · 2026-09-05

The check may have to happen in front of the origin. A robots.txt that permits a crawler does not help if a firewall or bot-protection layer answers that crawler with a challenge — a failure that is invisible in robots.txt and invisible in analytics. Perplexity’s documentation says so in as many words and lists firewall settings for it.

THE THREE JOBS A CRAWLER DOES

Search · user retrieval · training

The tables on this page are sorted by what an agent does, because that is what the controls attach to. The same provider often runs one of each. This three-way split is our framing — INFERRED from the documented purposes, not a taxonomy any provider publishes — and the rows below say, per provider, where the documentation actually separates the three and where it does not.

The table scrolls sideways.

SEARCH BOTS

The agent to allow if you want to appear in that provider’s answers

The table scrolls sideways.

OpenAI also documents a fourth agent, OAI-AdsBot, which belongs to none of the three classes: OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models. — DOCUMENTED · 2026-09-05 · OpenAI · crawler documentation.

Two entries that are absences

USER-RETRIEVAL AGENTS

One page, because a person asked — and robots.txt may not apply

These agents fetch a page because someone asked a question that needs it. Of the providers on this page that document one, five state a user-initiated exception to robots.txt — OpenAI, Perplexity, Google, Meta and Amazon, the last two in the five-provider table further down. Two document such an agent without stating an exception: Anthropic, which states blanket compliance, and Mistral. Whether that difference shows in practice is not something a table of documentation can answer — that would be OBSERVED, and it would need server logs. It is recorded as an asymmetry in the documentation, nothing more.

The table scrolls sideways.

• Not a permission. “May not apply” describes the provider’s behaviour. It does not make the fetch authorised, and it does not affect any other legal or contractual control you have.
• Not a search route. OpenAI: ChatGPT-User is not used to determine whether content may appear in Search. Please use OAI-SearchBot in robots.txt for managing Search opt outs and automatic crawl. Allowing ChatGPT-User does not get a page into ChatGPT search.

TRAINING BOTS

A data-licensing decision — separate from search where the provider documents it as separate

Blocking these is a decision about your content. With the three providers below, the documentation separates that decision from search visibility — twice in the provider’s own words, once as our inference — and the column says which is which.

The table scrolls sideways.

The most misread control on this page

Google-Extended does not govern AI Overviews or AI Mode. Those are Search, and Search is governed by Googlebot. Google-Extended governs Gemini model training and grounding in Gemini Apps and Vertex AI, and Google states that it does not affect a site’s inclusion in Search. That a site which disallows Google-Extended can still appear in Google’s AI answers follows from those two documented statements — INFERRED, not quoted.

Separate controls do not always mean separate crawls. OpenAI: If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling. — DOCUMENTED · 2026-09-05.

CONTROLS THAT ARE NOT USER AGENTS

Page level: a meta value, a Search Console setting

Not every control is a line in robots.txt. These work at page level, without any user agent, and four providers document one. Microsoft’s two tags separate three states: in the AI answer with content (no tag), in the answer as URL, title and snippet only (NOCACHE), out of the answer entirely (NOARCHIVE) — and Microsoft states that both tags keep the page in ordinary search results.

The table scrolls sideways.

FIVE MORE PROVIDERS

Mistral · Meta · Apple · Amazon · Common Crawl

Five providers that document their agents and are not in the tables above. Mistral documents a clean triple. Meta documents a search-class agent, a user fetcher and a training crawler — read from the rendered page, because the raw fetch was refused, and marked as such in every row. Apple’s search crawler also feeds its model training unless a second token is disallowed. Amazon documents three agents with independent settings. Common Crawl is neither a search engine nor an assistant.

The table scrolls sideways.

WHAT IS NOT DOCUMENTED

Two names that circulate, and no provider source for either

Two crawler names come up constantly and appear in no document by their provider. They are UNVERIFIED here, and they stay that way until the provider documents them. Third-party directories were found and refused as sources — several exist, and none is a provider source.

User-Agent: *
Allow: /

The table scrolls sideways.

Also not on this page

• A recommended robots.txt. What to allow is a business decision about your content, not a technical default. This page gives you the classes so you can make it.
• Any claim that allowing an agent produces a citation. Every provider that addresses the question declines to guarantee inclusion. Eligibility is not selection.
• Crawler lists compiled by third parties. Several exist; none is a provider source, and this page has one sourcing rule: a DOCUMENTED row comes from the provider.
• Server-log observations. We have not published crawler-hit logs. When we do, those rows will be OBSERVED and clearly separated from these.

LLMS.TXT

Google’s words, twice — and the file we maintain anyway

Google Search ignores it. In Google’s words:

You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them. Google · DOCUMENTED · 2026-09-05

It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them. Google · DOCUMENTED · 2026-09-05

That is the whole Google answer, and this page does not soften it. Which tools do read the file? Some do. This page does not list them, because we have not verified a single one from a provider source — any list here would be UNVERIFIED dressed as a reference. If we verify one, it gets a row, a link and a date like everything else.

We publish an llms.txt on this site and maintain it by hand, for the tools that read it. It is not part of any claim about Google.

Three neighbouring claims, same source, same date

The table scrolls sideways.

WHAT WE RUN ON OUR OWN SITE

Disclosure · OBSERVED · checkable against the live file

blackoarstudio.com/robots.txt · read 2026-09-05

User-agent: *
Allow: /

Sitemap: https://blackoarstudio.com/sitemap-index.xml

Every agent on this page is currently allowed on this site, including the training bots. That is a deliberate position for a studio with no licensed archive to protect, and it is not advice. A publisher with content worth licensing might reasonably decide the opposite — the training table above is the table that lets them do so without losing search visibility, where the provider documents the separation.

OBSERVED — a live capture of our own file, 2026-09-05. Not a provider statement.

RE-VERIFICATION SCHEDULE

Next: 2026-12-05

This page describes other people’s systems and will go out of date without warning — in a way a broken-link check will not catch. Our own source list was compiled on 26 August 2026 and re-read on 2026-09-05. What the re-read found — OBSERVED:

• Every URL on our own list of 26 August 2026 still resolved on 5 September 2026. A link check would have reported nothing.
• In the same ten days, two agents appeared in provider documentation that were on no list of ours: OpenAI’s OAI-AdsBot and Meta’s Meta-WebIndexer. Neither moved a URL.
• Google’s guide became explicit enough on llms.txt to quote verbatim. The URL did not change; the words did.
• Two documentation homes moved (OpenAI’s bots page and Google’s crawler overview), both with a redirect. Both are recorded under their current address here.

Content drifts under stable URLs. That is why every row carries a date rather than just a link.

• Every row carries the date its source was last read. That date is the claim.
• Re-verified quarterly, and whenever a provider announces a change.
• When a source moves, changes its wording or disappears, the row is marked and re-dated. The old claim is never silently edited — the same rule we apply to our own measurements.
• Next scheduled re-verification: 2026-12-05.

SOURCES

Provider documents only · read 2026-09-05 · preserved with checksums

• OpenAI · crawler documentation developers.openai.com/api/docs/bots
• Google · common crawlers developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Google · user-triggered fetchers developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
• Google · AI features and your website developers.google.com/search/docs/fundamentals/ai-optimization-guide
• Bing · which crawlers does Bing use www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 crawler table is JavaScript-rendered — read from the rendered page
• Bing Webmaster Blog · NOCACHE and NOARCHIVE blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat
• Perplexity · crawlers docs.perplexity.ai/docs/resources/perplexity-crawlers
• Anthropic · does Anthropic crawl data from the web support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Mistral · robots docs.mistral.ai/robots
• Meta · web crawlers developers.facebook.com/docs/sharing/webmasters/web-crawlers/ raw fetch refused (HTTP 400) — read from the rendered page
• Apple · about Applebot support.apple.com/en-us/119829
• Amazon · Amazonbot developer.amazon.com/amazonbot
• Common Crawl · CCBot commoncrawl.org/ccbot

Two documents could not be captured raw and are marked above and in their rows. Both were read from the rendered page. Recorded so a reader knows which rows rest on a preserved document and which on a transcription. The IP endpoints in the verification table were fetched on the same date and preserved alongside the documents.