CRAWLER LOG
01.

THE SITE AND THE WINDOW

88 pages · two languages · 11,701 verified requests

blackoarstudio.com is a young studio site: domain live since July 2026, 88 pages in English and German that existed for the whole window, a sitemap, and a robots.txt that allows every agent. It runs behind Cloudflare on the free plan. Bot Fight Mode is off, and Cloudflare’s AI crawler controls are set to allow. Rate limiting and Cloudflare’s managed rules are on — finding 7 shows whom they stopped.

Two demo projects live on subdomains of the same zone. Both carry noindex and have no sitemap. They are part of the record because crawlers visited them.

HOW TO READ THESE NUMBERS

Source · what “verified” means · sampling · classes

Source: Cloudflare’s GraphQL Analytics (the httpRequestsAdaptiveGroups dataset), one query per day, read on 14 October 2026. No log line was edited or removed.

Verified: a request counts when Cloudflare files it under a verified-bot category. The classification — the names in the category column included — is Cloudflare’s, not ours. A request that only carries a crawler’s name in its user agent is not verified; those are counted separately below.

Sampling: the dataset is adaptive. At higher volume Cloudflare counts a share of requests and extrapolates. In this window the highest daily average sample interval was 1.56 — close to counting every request, but the numbers are Cloudflare’s estimates, not raw log lines.

Every number on this page is OBSERVED. Every conclusion we draw from them is marked INFERRED and stands apart from the number. Where we know nothing about an agent, it says UNVERIFIED. The classes are defined once, on the methodology page.

WHO CAME

Search and AI agents with more than five requests · two SEO tools for comparison

Verified bot requests to all hosts of the zone, 16 September 2026 – 13 October 2026. “Pages reached”: distinct sitemap pages answered 200 or 304 on blackoarstudio.com, of 88.

The table scrolls sideways.

Not in the table: page previews (Facebook, X, LinkedIn, Pinterest), monitoring services and Lighthouse runs. They are part of the 11,701 verified requests. OBSERVED

WHAT THE LOG SHOWS

Eight findings · number first, conclusion after

1 · Reading every page is the exception

Of 88 pages, one AI crawler reached all of them: PetalBot, 88. Googlebot reached 75, ClaudeBot 63, bingbot 52, Amazonbot 45, GPTBot 40, OAI-SearchBot 31, Applebot 26. The two SEO tools reached 87 each. OBSERVED

On a site this size, a page that exists is not a page each crawler read in a given month. Whether a crawler had read a page before the window, this log cannot say. INFERRED

2 · ClaudeBot checked daily and read on two days

ClaudeBot made 1,379 requests and was active on all 28 days. 606 went to robots.txt and 621 to sitemaps. It requested pages on blackoarstudio.com on two days only: 97 on 3 October 2026, 2 on 4 October 2026. 374 of its requests asked the two demo subdomains for a /sitemap.xml that does not exist there. OBSERVED

The pattern reads as frequent checking and reading in bursts. Why it schedules this way is not documented here, and we do not guess. INFERRED

3 · No AI agent asked for llms.txt

In 28 days, verified AI crawlers and AI assistants requested /llms.txt 0 times. The file was requested 26 times by Lighthouse runs and 46 times by unverified clients — curl, browsers, scripts. OBSERVED

On this site, in this window, the file was read by tools that check whether it exists, not by the agents it is written for. We keep it — for the tools that do read it — and we do not count it as a reason anything was crawled or cited. INFERRED

4 · Fetches a person started

ChatGPT-User made 185 requests on 27 days, 83 of them for the homepage, and reached 43 pages. Claude-User made 213 on 22 days and reached 33 — the Shopify health check, pricing, and the audit and QA pages most often. Claude-User requested robots.txt 49 times; ChatGPT-User 0 times. OBSERVED

Both providers document these agents as acting on a user’s request — see the crawler reference. A request is not a visitor and not an enquiry: the log shows that the page was fetched, not what the person was told. INFERRED

5 · Our own mistake: redirects on internal links

79 of Claude-User’s 213 requests were redirects — it asked for pages without the trailing slash and was sent on to the slashed address. bingbot was redirected 357 times, PetalBot 388, Googlebot 251. The cause was ours: 3,839 internal links on this site — counted when we fixed them — pointed at the slashless form. OBSERVED

Fixed on 14 October 2026, after this window: internal links now point at the address each page actually has. The next edition shows whether the redirect counts fall.

6 · noindex does not stop requests

The two demo subdomains carry noindex and have no sitemap. Verified bots still made these requests to them: OBSERVED

• ClaudeBot 759
• bingbot 171
• OAI-SearchBot 131
• GPTBot 89
• Amazonbot 68

noindex is an instruction about the index. It is not an instruction to stay away, and the log treats it as such. INFERRED

7 · A crawler name is a claim

1,775 requests carried the name of a known crawler in their user agent without being verified. 1,075 of them asked for paths like /.env, /.git or /wp-config.php — files that hold credentials on other sites and do not exist on this one. Most-claimed names: PerplexityBot 341, ClaudeBot 306, Googlebot 255. 55 requests were blocked by rate limiting or managed rules — all of them unverified. Verified bots blocked: 0. OBSERVED

A user agent is a sentence anyone can write. How each provider lets you check a request is in the reference, under verification. Counting by user agent alone would have added these requests to the crawler numbers above. INFERRED

8 · Agents that came once

ShapBot made 527 requests on a single day, 6 October 2026, and reached 82 pages. Reflectionbot made 110 over 13 days. Cloudflare files ShapBot as AI Search and Reflectionbot as AI Crawler. OBSERVED

We have not read documentation from either operator. What they collect for and where it ends up is unknown to us. UNVERIFIED

WHAT WE CHANGED — AND WHAT WE DID NOT

Dated · on this site only

• 16 September 2026, before the window: /sitemap.xml on blackoarstudio.com answers with a 301 to the sitemap index. In this window, every /sitemap.xml that ClaudeBot got a 404 for was on a demo subdomain, where the redirect does not apply (finding 2).
• 14 October 2026, after the window: internal links carry the trailing slash (finding 5); the contact page is linked from every page instead of only through a pop-up.
• Not changed: robots.txt, Cloudflare’s crawler controls, the demo subdomains. They stay noindex and get no sitemap — the requests to them are recorded, not answered with a fix that would invite more.

WHAT THIS DOES NOT SHOW

• How these crawlers behave on other sites. One site with 88 pages and low traffic is a small sample.
• Whether any request led to an index entry, a citation, a recommendation or a customer.
• Why an agent did what it did. The log records requests, not intentions.
• Where the unverified requests came from: network data is not available on this Cloudflare plan.
• Agents Cloudflare does not verify. A genuine crawler without verification sits in the unverified count.

NEXT EDITION

Edition 02 is planned for 14 November 2026: the same queries, the same table, the same classes — published whatever it shows. This edition stays as it is.