39% of hotel websites are missing from Common Crawl

There are two ways an AI can know about your hotel. It can fetch you live at query time (retrieval — a search, scraping or not, plus Google Places, OTAs, reviews), or it can already know you from training. The open-web part of that training layer often starts with Common Crawl, the monthly web archive many LLM builders use. If your own site isn’t in the crawl, a model can’t learn your hotel from that source — though it may still pick you up indirectly through OTAs, reviews and travel guides. I hadn’t seen anyone look at this for hotels specifically — so I dug into it.

Two things to keep straight before the numbers. First, being in Common Crawl isn’t the same as being in a model’s final training set — that data gets filtered, deduplicated and down-weighted. But within this source, absence is decisive: a page that was never captured can’t survive any later filtering. Second, the point isn’t that AI can’t find these hotels at all. It’s that the hotel’s own website is missing from one of the main open-web memory layers, leaving OTAs, directories and review platforms to define the property instead.

Who’s in, who’s out

A representative sample of the hotels we checked. Green dots are in Common Crawl; red are absent. The pattern is geographic — coverage is denser in some markets than others.

How many pages — and why that matters less here

We counted how many pages of each hotel the May 2026 crawl captured. A caveat before reading this chart: a hotel website is a small site. Home, a rooms page, a few room types, a gallery, contact, maybe a restaurant — that’s often the whole thing. The median hotel that’s in the crawl is captured at just 9 pages, and 9 pages can be the entire site. So unlike a news or e-commerce domain, depth isn’t really the worry for a hotel — presence is. The line that matters is 0 vs. 1+ — or rather, that’s the first line. The second is whether those captured pages carry real hotel content and not just a JavaScript shell (more on rendering below).

The ~3,400 hotels (3%) captured at 100+ pages are mostly not small properties at all — they’re hotels whose listed website sits on a big shared platform (a chain domain, an OTA, a directory). For an ordinary independent on its own domain, 5–30 captured pages is a complete reading, not a shallow one. Which is why the real story below isn’t depth — it’s the 39% at zero.

“Chains” underperform — but it’s really one budget group

The headline split looks backwards: independent hotels are in Common Crawl far more often than chain properties. But that average hides almost everything interesting, so we broke it down by brand.

Coverage by brand

Green = above the 60.6% overall line. Brands with ≥15 domains in the run.

What absence actually looks like

Six real rows from the run. The two at the top are recognisable independents with their own sites — and they’re simply not in the crawl. The middle two are budget-chain microsites at zero. The last two are small hotels captured at a page or two.

A reviewed, bookable hotel whose official site is at 0 is invisible through that site to any model learning from this archive — it can only be reached live, if the engine runs a search.

By country and TLD

Coverage varies sharply by market. Among the largest markets in the dataset:

One distinction first, to avoid a false contradiction: country is the hotel’s location; TLD is the website’s domain. Spain reads 58.6% by country yet 37% on the .es TLD, because many Spanish hotels sit on .com — the two measure different things.

With that in mind, the TLD tells a sharper story about how the open-web training layer may under-represent non-English markets. Local European TLDs are present but shallow: a .de hotel is well-crawled (71%) but at only ~39 pages on average, where a .com hotel averages ~109. And .es is the clear laggard at 37% — Spanish hotels on .com do fine, but the .es TLD itself is poorly crawled.

The crawl knows local-market hotels — but thinly. This looks like one mechanism behind the English-leaning tilt we see in live AI answers — showing up a layer earlier, in the training data itself.

Why some hotels are missing

Absence isn’t random. Three things keep a hotel out of the crawl:

• Rendering. If your site only assembles its content after JavaScript runs, the crawler often captures an empty shell. Static HTML gets read; SPA booking widgets frequently don’t.
• Access. A CDN or firewall that blocks bots — often a default no one chose — turns the crawler away before it reaches a page. This is invisible in robots.txt, which is why our AI-blocking study’s ~3% figure is a floor.
• Connectivity — and this is the main lever. Common Crawl doesn’t crawl the web evenly. It decides which domains to capture, and how deeply, by Harmonic Centrality — a link-graph score for how well-connected a domain is. High-scoring domains get crawled often and deep; low-scoring long-tail sites get crawled rarely or not at all (documented in Mozilla’s “Training Data for the Price of a Sandwich”). For a hotel this is usually decisive: a property with few inbound links scores low and gets skipped, however good the site is. Metehan Yeşilyurt’s work on Common Crawl rank lays this out, and you can look up your domain’s Harmonic Centrality and PageRank at webgraph.metehan.ai.

Method & limits

We took every hotel in our 200K-property index with its own website, a Google Place ID, and at least 10 reviews — 142,405 hotels, resolving to 108,109 distinct domains(some chains consolidate many properties under fewer domains). Junk (parked domains, closed properties, non-hotels) was filtered out. Each domain was checked against the columnar URL index of the May 2026 Common Crawl snapshot (CC-MAIN-2026-21), counting captured pages for the host and its www variant. Captured = 1+ page (what we count as “in Common Crawl”); deep capture = 5+ pages; shallow = 1–4; absent = 0.

FAQ