All figures are first-party, from v_ai_bot_requests_projects and my own Vercel logs. One property is not the industry — treat the shape as the finding, not the absolute numbers. If you want the flip side of this coin (what AI crawling gives back), that lives in measuring AI hotel traffic.
How hard they crawl
Two sites sit behind these logs. One is a real Paris hotel (I’ll call it its project key, hotel-ranque). The other is hotelrank-lp, the hotelrank.ai marketing landing page — a page about hotels, not a hotel. Between them, AI bots have made 119,609 logged requests: 6,378 against the hotel and 113,231 against the landing page.
The monthly curve climbs steeply on both. The landing page went from roughly two thousand AI-bot hits in December to more than thirty thousand a month by late spring. The hotel is an order of magnitude smaller but rises the same way. Watch the two axes, though: the landing page is on the right and dwarfs the hotel on the left.
Monthly AI-bot request counts. Left axis: the hotel. Right axis: the landing page. July is a partial month (through the 24th).
The early ramp is partly a measurement artifact
Request logging was switched on around the middle of March 2026. Everything before it — December, January, February — is an undercount, not a calm baseline. So read the steep-looking early climb with suspicion: some of it is real growth in AI crawling, and some of it is simply the logger catching up with traffic that was already there.
Who’s crawling
On the hotel, the field is not crowded. OpenAI accounts for 49.6% of AI-bot requests and Claude for 31.8% — a two-name duopoly worth 81.4% of the crawl between them. Apple (6.4%), Amazon (4.3%), and Meta (2.6%) trail well behind, and everyone else is a rounding error. GPTBot and ClaudeBot, in other words, are most of what visits a small hotel’s website now — the two of them show up more than every other AI company combined.
Share of voice by brand on the hotel. OpenAI and Claude together are 81.4% of the crawl.
The landing page tells a different story. OpenAI still leads at 42.8%, but the number two is ByteDance at 32.7% — Bytespider hammers the marketing page far harder than it bothers with the hotel, where it sits at under 2%. Claude, dominant on the hotel, is only 7.9% here. Same web, two very different crawl diets, depending on whether the page reads as a business someone might book or as content someone might scrape.
Share of voice by brand on the landing page. ByteDance climbs to number two at 32.7%.
Reminder on the two sites: hotel-ranque is the actual hotel; hotelrank-lp is a marketing landing page, not a property. If you plan to block a crawler outright, see the robots.txt & AI-blocking study first — who you shut out matters.
Three kinds of crawl
Not every bot hit means the same thing, and lumping them together is how people end up either panicking or shrugging when they should do the opposite. The logs split into three intents.
Corpus crawlers stocking a training set — GPTBot, ClaudeBot, Bytespider, AmazonBot, CCBot. On the hotel this is 3,219 requests (50.5%): the single biggest slice, and the one that gives back the least in the short term.
Indexers building an AI-search surface — OAI-SearchBot, PerplexityBot, Applebot. 1,854 requests (29.1%) on the hotel. These can put you in front of a searcher, so they earn more patience than pure training crawlers.
On-demand fetches — ChatGPT-User, Claude-User — a bot pulling your page live because a person just asked about you. 1,305 requests (20.5%) on the hotel, and the most valuable of the three.
That last kind is the interesting one. An on-demand fetch means a human is, at that moment, asking an AI something your page can answer — which makes the on-demand curve the closest thing to a server-side AI impression counter you can get without any tracking script. On the hotel it went from 7 such fetches in December to 389 in June (with 312 already in a half-finished July). Whatever else the crawl costs, this line is the part that means someone was actually asking.
On-demand fetches on the hotel — a fetch per real question asked. July is partial.
Crawl versus referral
The obvious next question is whether all this crawling pays its way in visitors. I can only answer it for a single month, and even then barely. Referral logging — humans arriving from an AI product — was only ever captured for a December 2025 window. In that window the hotel received 86 visits attributed to Google, 4 to OpenAI, 3 to Perplexity, and 2 to Brave.
In that same December, OpenAI crawled the hotel 126 times and sent 4 people. Tempting as it is to write that as a ratio and wave it around, I won’t: the referral logging didn’t continue past December, and one month against a barely-warmed-up logger proves almost nothing about today. So this is a single directional data point, not a rate.
Do not compute a current crawl-to-visit ratio from this.
One month, referral logging since discontinued, during the period when request logging itself was still incomplete. It tells you the direction — a lot of fetching for a little referring — and nothing reliable about the size of the gap now. Referrals are a whole separate measurement problem, and I treat them as one in the traffic-measurement piece.
The hidden cost: every crawl of a dynamic page is database work
Bandwidth is the cost everyone talks about. It is not the one that got me. The one that got me was a different project on the same stack — a set of programmatic race-guide pages — and the culprit wasn’t even an AI bot.
To be clear up front: this worked example is Googlebot, a search crawler, not an AI bot. I use it because it shows the mechanism cleanly — and then I bridge, because AI bots do exactly the same thing to the same database.
In one twelve-hour stretch, Googlebot Smartphone made 33,329 requests from 37 IP addresses — about 46 a minute, sustained — against a set of programmatic pages. For scale, the next-busiest user-agent over the same window managed 1,094. Three race apps took the brunt: marathon-de-paris-2026 alone logged 31,247 requests over 24 hours across just 129 URLs (roughly 240 fetches per page), with letape-du-tour-2026 (17,755) and semi-de-paris-2026 (16,630) close behind. Add them up and it’s about 65,000 server-rendered hits in a day across only ~334 pages — every one of those URLs fetched on the order of two hundred times. The peak minute hit 1,122 requests.
Here is where the real cost hid. Each of those requests hit a server-rendered page — so each one was database work, not a static file handed back by a CDN. And worse, I had wired a Vercel→Supabase log drain that wrote one row per request, including the full raw JSON payload. Running since December 2025 and never pruned, that table — vercel_logs — had grown to 2.37 million rows and 5.6 GB, absorbing around 92,000 writes a day. The tooling I’d added to watch the bots had quietly become more expensive than the bots themselves.
I was, in effect, renting a warehouse to keep a signed receipt for every visitor who walked in, looked around, and bought nothing. The fix was unglamorous: I filtered the drain down to AI-bot traffic only and pruned the history, which took the table from 5.6 GB to 952 MB, and kept a tiny hourly crawler_hits rollup so that a sudden bot spike is still visible without storing every last row.
The bridge to AI bots
Googlebot is a search crawler, but a fleet of AI crawlers — GPTBot, ClaudeBot, Bytespider — behaves identically against a dynamic site: every fetch of a server-rendered URL is a database read, and if you log them you pay to store the record too. The same Supabase project now isolates the AI bots in that lean crawler_hits rollup, which is where the 119,609-request dataset at the top of this page comes from.
What it costs, roughly
I want to give you a bandwidth number, and I can’t give you a real one: the logs record requests, not bytes. So the only honest figure is an explicit back-of-envelope — the combined 119,609 requests multiplied by an assumed average page weight. Change the assumption and the answer changes; that is the whole point of showing three of them.
| Assumed avg page weight | Typical of | ≈ Transfer over the period |
|---|---|---|
| 200 KB | lean, cached HTML | ~22.8 GB |
| 500 KB | HTML + a little JSON | ~57.0 GB |
| 1 MB | image-heavy page | ~116.8 GB |
So: somewhere between roughly twenty and a hundred-and-twenty gigabytes of transfer over about seven months, entirely dependent on a page-weight number I made up. On modern hosting that bandwidth is cheap — often free at this scale. Which is exactly why the storage-and-compute story in the previous section matters more: the bytes on the wire were never the expensive part. The expensive part was turning every crawl into database work and then paying again to log it.
What to do about it
None of this argues for slamming the door. It argues for triage — bounding the reads, bounding the writes, and being deliberate about which bots you serve generously and which you make work for it.
Cache and pre-render to bound the reads
The 200-fetches-per-URL problem only bites because each fetch re-rendered a page from the database. Serve those pages from cache or ISR and a bot fetching the same URL two hundred times costs you one render, not two hundred. This is the single highest-leverage fix for a programmatic site.
Filter and aggregate your logging to bound the writes
A row per request with the full payload is what grew my table to 5.6 GB. Log the bot traffic you care about, drop the rest, and keep a cheap rollup — an hourly count per bot — so a spike is still obvious without warehousing every request. You want to see the weather, not archive every raindrop.
Put a CDN in front
A CDN edge absorbs repeat fetches of the same asset before they ever reach your origin or your database. For crawl traffic — which is repetitive by nature — this quietly removes most of the load you were about to pay for.
Triage by intent, not by reflex
This is where the three-kinds split earns its keep. Serve ai_search and ai_user generously — they can put a guest in front of you. Rate-limit or challenge the pure ai_training crawlers if they get greedy. But go easy on the blunt instrument: blanket-blocking a crawler can also cut you out of live AI discovery, so decide with the blocking study open, not on a hunch.
And a word on measurement, since it started this whole mess: the instinct to log everything so you can “see what the bots are doing” is the instinct that cost the most. A visibility score built on that habit is easy to inflate and easy to over-read — I unpack why a fetch is not a booking in the visibility-scores guide.
Methodology & honest limits
- Dataset. Request-level AI-bot logs from
public.v_ai_bot_requests_projects(Supabase project rjpaprlepwapbtelqzrx), December 12 2025 → July 24 2026, for two project keys:hotel-ranque(a real hotel) andhotelrank-lp(a marketing landing page). - Monitoring, not blocking. Vercel logging ran in “Log” mode — it observes, it does not block. A row with
status -1orsource=firewallwas logged, not refused. None of these bots were turned away. - The logger started mid-March. December through February are undercounts, so the early part of every curve understates reality — the ramp is partly the logger waking up, not pure growth.
- No bytes. The logs count requests, not transferred bytes. Every bandwidth figure here is an explicit assumption (requests × an average page weight I chose), never a measurement.
- Googlebot is a search crawler. The 5.6 GB worked example is Googlebot, not an AI bot. It illustrates the dynamic-page-equals-database-work mechanism; the bridge to AI crawlers is argued, not measured, in that section.
- One illustrative property; July is partial. A single hotel is not the industry — the shape is the point, not the absolute numbers — and July 2026 counts run only through the 24th.
Summarize with AI
