Measured data
Which AI crawlers actually visit a Houston business site
Everyone has an opinion about AI search. This is a count. Every figure below comes from this site's own request log, updated daily.
Which AI crawlers visit a small business website?
Across 5,146 logged requests to this site, 14 distinct AI agents appeared. The largest share came to build a search index or harvest training data. Only 22 requests were an assistant fetching a page to answer someone's question in the moment, which is the only category tied to a real person waiting on an answer.
- Total AI requests
- 5,146
- Distinct agents
- 14
- Last 30 days
- 2,306
since 22 May 2026
named crawlers, not counting search engines
requests in the current window
Why they came
The useful split is not bot versus human, it is purpose. Every major operator runs separate agents for these three jobs, and the names differ by one word, so a report that lumps them together tells you nothing.
| Purpose | What it means | Requests | Share |
|---|---|---|---|
| retrieval | fetched to answer a live question | 22 | 0.4% |
| search | indexed for an assistant to query later | 2,673 | 51.9% |
| training | harvested for a training corpus | 2,451 | 47.6% |
Retrieval is the number worth watching. It is the only one where a person is waiting on the answer. It is also, on this site, by far the smallest — which is worth knowing before paying anyone for AI search work.
Every agent seen, by volume
| Agent | Operator | Purpose | Requests |
|---|---|---|---|
| Amazonbot | Amazon | search | 2,070 |
| ClaudeBot | Anthropic | training | 1,210 |
| GPTBot | OpenAI | training | 632 |
| CCBot | Common Crawl | training | 605 |
| Applebot | Apple | search | 582 |
| ChatGPT-User | OpenAI | retrieval | 14 |
| PerplexityBot | Perplexity | search | 10 |
| OAI-SearchBot | OpenAI | search | 8 |
| Claude-User | Anthropic | retrieval | 7 |
| Google-Extended | training | 3 | |
| Claude-SearchBot | Anthropic | search | 2 |
| Bytespider | ByteDance | training | 1 |
| DuckAssistBot | DuckDuckGo | retrieval | 1 |
| YouBot | You.com | search | 1 |
How this was measured
Every successful page request to this site is logged server-side with its user-agent string. A request counts as an AI agent when that string contains a token published by the operator — GPTBot, ClaudeBot, PerplexityBot and so on. Matching runs most-specific first, because Claude-User and ClaudeBot are different agents with different jobs, and so are Applebot and Applebot-Extended.
Ordinary search crawlers are excluded. Bingbot, AhrefsBot and the rest are not AI agents and counting them would inflate every figure here.
What this cannot tell you. A user-agent string is self-reported and can be forged, so these are claimed identities. And a fetch is not a citation: knowing an assistant retrieved a page is not evidence it used it in an answer, or named the source. Anyone selling you a guaranteed AI citation cannot measure it either.
Figures regenerate daily. Last updated 26 August 2026.
Want your site readable by these agents?
Making a page an assistant can actually use is a specific piece of work, and it is measurable. Call James and ask what it would involve for your site.