On 29 August, Konstantin Ryabitsev published what AI crawlers cost kernel.org: six million requests a day to git.kernel.org, and “14 CPU cores doing nothing but rendering git commits as html” across five servers, roughly a fifth of the ninety cores available.
The number everyone will quote is 98 percent. The sentence next to it is the one worth keeping:
“It’s impossible to tell with certainty which of these are bots and which are real humans.”
That is an operator with full access to his own logs saying he cannot establish who is asking. We measured the same layer from the other end, on a much smaller site, and found something narrower and worse.
What our logs say
Across 2,418,982 log lines over two weeks, we took every request carrying an AI crawler identity and checked each line against the IP ranges the vendor publishes for it.
checkable AI identities 19,582 lines
verified against vendor 17,547 90%
forged 2,035 10.4%Per vendor the spread is wide. GPTBot and ChatGPT-User came back 97 percent genuine. OAI-SearchBot 80 percent. PerplexityBot 87 percent. Perplexity-User was 78 lines, of which zero verified.
The forgeries are not mislabelled infrastructure. The obvious objection is that published IP lists go stale, so an unlisted address might simply be a new vendor range. We checked the twelve worst offenders by what they requested rather than where they came from. All twelve were hunting credentials: /.env in six variants, /.git/config, /.aws/credentials, /service-account.json, /firebase-adminsdk.json, and a directory traversal aimed at /etc/passwd. One address presented thirteen different crawler identities in thirty-one hours, among them Claude-User, GPTBot, ChatGPT-User, Perplexity-User, CCBot, DeepSeekBot, TelegramBot and Amazonbot.
The identity that cannot be checked at all
Here is the finding that changes what the other numbers mean.
ClaudeBot 17,544 lines
Claude-User 346
Claude-SearchBot 34ClaudeBot is the single most frequent AI identity in the logs, more frequent than every OpenAI and Perplexity identity combined. It is also the only one that neither of the two standard checks can settle.
The first check is the published IP range. OpenAI publishes ranges for GPTBot, OAI-SearchBot and ChatGPT-User; Perplexity publishes them for both of its agents; the documented path for Anthropic’s list answers with a redirect to the documentation home page.
The second check is reverse DNS, and it fails for a subtler reason. ClaudeBot requests do carry a PTR record. On a server we operate, twelve of twelve checked resolved, every one of them into the address space of a cloud provider. A reverse DNS record proves something only when the operator controls the zone: Applebot arrives with a PTR under applebot.apple.com, and Apple controls that name. Anyone can rent a machine in a cloud, so a PTR pointing at one names the data centre and not the sender.
This is not a claim that those requests are forged. We did not measure that, and it cannot be measured without the list. The finding is exactly this: the identity that appears most often is the one an operator has no way to test. Every rule written against it, whether a rate limit, an allow or a block, is applied to a name that anyone can type.
The third measurement, and why it belongs here
If crawler identity were reliable, the polite instruments built on top of it would work. One of them is llms.txt, a file that offers language models a curated list of a site’s useful pages.
On one of the sites we run, over two weeks and 276,396 log lines:
requests carrying an AI crawler identity 3,195
fetches of /llms.txt (excluding our own) 24
of those, carrying an AI identity 0Both filters demonstrably work: the same query finds our own 26 test fetches, and 353 AI-identified requests for /robots.txt in the same window. The file is fetched, just not by the systems it addresses. The 24 fetches came from SEO tools (PoweredByBot, PipericBot, Dataprovider), plus curl and browsers.
That has a second-order effect worth naming. The tools that measure llms.txt adoption generate the traffic their users then read as adoption.
What this does not say
It does not say 98 percent of kernel.org’s traffic is bots. Ryabitsev estimates it and says plainly that certainty is not available. Quote the sentence, not the number.
It does not say Anthropic’s crawler misbehaves. It says its requests cannot be verified by an operator, because the list required to do so is not published. Those are different statements, and only one is measured.
It does not say Anthropic crawls from a cloud provider. The reverse DNS records we saw point there, and the opposite reading is at least as plausible: that we saw no genuine ClaudeBot at all. Either way the identity could not be settled, and that is the finding.
It does not say llms.txt is ignored everywhere. It says that on the site measured, over the two weeks covered, no request carrying an AI identity fetched it. The file on that site is byte-identical to one we run elsewhere, so the two measurements are independent in their traffic and not in the file.
Ten point four percent is a share of the checkable requests, not of AI crawler traffic overall, and it is not a constant. It covers the fortnight to 28 August on the first site. We ran the same check on a second site we operate and got 59 percent over the fortnight to 25 August, then 69.2 percent over the fortnight to 31 August. Two measurements, two windows, ten points apart on the same site. That is not a contradiction, and the movement is the useful part: over the earlier window, two addresses alone made 1,626 requests under five crawler identities at once, so one pair of scanners set the rate. On that same data, counting distinct addresses instead of requests moved the result by 44 points: 85 percent genuine by address, 41 percent by request.
Set the two sites side by side in absolute numbers and the reason is plain. Take the later of the two windows on the second site, because it is the one measured with the same tooling as the first: the forged requests are of a similar order on both, 2,035 against 2,355. The verified ones are not: 17,547 against 1,048, a factor of seventeen. The percentage is therefore mostly a statement about the denominator. Scanners knock at roughly the same rate everywhere; genuine crawlers go where the content is. A smaller site does not attract less forgery, it attracts less of everything else, which means its share runs higher than a headline figure taken from a busier one.
So the figure describes who happened to be scanning a particular site in a particular fortnight, counted a particular way. It is not a property of AI crawler traffic, and no single number from one site should be quoted as one. The uncheckable requests remain the larger group in both measurements.
What to do with it
Stop treating the user agent as an identity. It is a claim, and on this evidence one in ten checkable claims was false. Where a vendor publishes ranges, verify against them; the check costs one lookup.
Where no list exists, check the machine instead of the name. On one server, GPTBot, Applebot, Amazonbot, Claude-User and Claude-SearchBot all arrived from the same three cloud addresses. Five identities, three machines. An address claiming several vendors at once is proof of forgery that needs no vendor list, which makes it the only check that also works for the identities nobody publishes ranges for.
Ask your vendors for the list, not for a promise. The practical difference between the three major crawlers today is not how they behave. It is whether an operator can tell that it was really them.
Watch what the failing requests ask for. Every forged identity we examined was looking for credentials. A fake crawler identity is not a traffic problem wearing a costume, it is a credential scan that chose a costume your logs are inclined to trust.
Note which defences do not depend on identity. kernel.org reached for proof-of-work, which never asks who is calling and instead makes the call cost something. Two thirds of the requests fail that challenge. That is the shift the two measurements point at together: instruments that ask who are you have stopped separating anything, and instruments that impose a cost still do.
Sources
- Konstantin Ryabitsev, “Creepy crawlies”, 29 August 2026: https://people.kernel.org/monsieuricon/creepy-crawlies (retrieved 31 August 2026). Source of the six million daily requests to git.kernel.org, the estimated 98 percent scraper share, the quoted sentence on the impossibility of telling bots from humans with certainty, the “14 CPU cores” quotation and the roughly one fifth of ninety cores across five servers, and the two thirds of requests that fail the proof-of-work challenge. The 98 percent figure is described by the author as an estimate and is reproduced here as one.
- Our own measurement, carried out 28 August 2026 over 2,418,982 log lines covering 14 to 28 August 2026: the 19,582 checkable AI-identified requests, the 17,547 verified against published vendor ranges and the 2,035 that were not, the per-vendor rates, the twelve source addresses examined by requested path, the thirteen identities presented from one address within thirty-one hours, and the 17,544 ClaudeBot lines. Vendor ranges used: openai.com/gptbot.json (21 prefixes), searchbot.json (35), chatgpt-user.json (204), perplexity.com/perplexitybot.json (8), perplexity-user.json (4). Each list was applied with a positive control confirming the matcher worked; the documented Anthropic list resolves to a redirect rather than a file, which is why no ClaudeBot line is counted as either verified or forged.
- The 59 percent figure comes from an independent measurement on the second site we operate, carried out 25 August 2026 over a fourteen-day window with the same published vendor ranges: 2,452 checkable requests, 1,003 verified, 1,449 not. The 44-point spread between counting addresses and counting requests is recorded in that measurement, as is the pair of addresses presenting five crawler identities at once.
- The 69.2 percent figure comes from a third measurement, on that same second site, carried out 31 August 2026 over 296,254 log lines covering 17 to 31 August 2026, using the prefix lists and the line-by-line check that produced the 10.4 percent figure on the first site, with all five matcher positive controls passing: 3,403 checkable requests, 1,048 verified, 2,355 not. It is quoted here alongside the 59 percent rather than in place of it, because the two windows overlap only in part and the supporting detail above (the 1,626 requests from two addresses, the 44-point address-versus-request spread) belongs to the earlier one.
- The reverse DNS check on ClaudeBot was carried out 31 August 2026 on a server we operate: 2,030 requests from 15 addresses, twelve of twelve checked resolving into a cloud provider’s address space, with Claude-User and Claude-SearchBot showing the same pattern. Applebot in the same log resolves under a name Apple controls, which is what makes that record evidence and the other one not. The measurement covers one server and one window, and a large share of its traffic is demonstrably forged; it establishes that the identity cannot be settled, not where Anthropic crawls from.
- The llms.txt measurement covers 276,396 log lines over 14 to 28 August 2026 on a site we operate, with two positive controls: our own 26 test fetches were found by the same filter, and 353 AI-identified requests for /robots.txt fell in the same window. An earlier run of this measurement produced 46 fetches and 334 AI requests; both figures were wrong, caught by the positive control, and are reported here only in their corrected form.