RESOURCES / HOSPITALITY / WHAT AI CRAWLERS SEE
What crawlers actually extract from your site.
We ran a test last month that has become standard in every hotel engagement. We took a luxury resort's website, disabled JavaScript in the browser, and reloaded the homepage. The full-screen video disappeared. The room carousel vanished. The booking widget was gone. The restaurant menus, spa services, event spaces, all of it, loaded via JavaScript after the initial page render. What remained was a logo, a navigation bar, and a single paragraph of text that said "Experience luxury redefined."
That stripped-down shell is what GPTBot, ClaudeBot, and PerplexityBot see when they visit. And it's the version that determines whether AI recommends this property or skips it entirely.
The JavaScript blind spot in hospitality
Hotel websites are among the most JavaScript-dependent on the web. Room availability widgets, pricing displays, photo galleries, virtual tours, interactive maps, and booking engines all rely on client-side JavaScript. For human visitors, the experience is polished. For AI crawlers, most of that content doesn't exist.
This isn't theoretical. An analysis of over 500 million GPTBot requests found zero evidence of JavaScript execution, even when the crawler downloaded JS files. AI crawlers fetch raw HTML on the initial request. They don't scroll, click, or wait for API calls. If content isn't in that first response, it's invisible.
The hospitality industry is particularly exposed because the elements AI needs most are often the ones hidden behind JavaScript: room types and descriptions, amenity lists, location details, pricing structures, policies, and FAQ content. These are exactly the facts a traveler asks about when they prompt ChatGPT with "best hotel near downtown Portland with a pool and free parking."
Hospitality consultant Ira Vouk analyzed several major hotel chain websites and found many were blocking most AI bots in their robots.txt files, making them practically invisible on AI platforms. OTAs, by contrast, are wide open to crawlers, with structured, crawlable data that AI can easily parse. This is a significant factor in why OTAs account for 55% of AI hotel citations while hotel websites account for just 13.6%.
What AI needs to find in your HTML
The fix doesn't require a complete website rebuild. It requires ensuring that the information AI needs is present in the server-rendered HTML, regardless of what JavaScript adds after the page loads.
Property fundamentals. Your hotel name, full address, phone number, star rating, and a clear description of what makes the property distinctive should be in plain HTML text on every key page. Not embedded in images. Not loaded via API calls.
Room and amenity details. Room types, bed configurations, key amenities (pool, spa, parking, restaurant, Wi-Fi), and accessibility features need to be in the source code. If a traveler asks AI about your property and the answer is vague or incomplete, this is usually why.
FAQ content. The questions your front desk answers twenty times a day are the same questions travelers ask AI. Check-in time, parking availability, pet policies, distance to the airport, breakfast hours. Build these into a structured FAQ section using real HTML, and mark it up with FAQPage schema so AI models can parse it directly.
Schema markup. Hotel, LocalBusiness, and LodgingBusiness schema types tell AI exactly what kind of entity you are. Add Review schema from your own testimonials. Include geo-coordinates, price ranges, and amenity lists in structured data format. If your schema is injected via Google Tag Manager or client-side JavaScript, AI crawlers won't see it. It must be in the server-rendered HTML.
The robots.txt check most hotels skip
Before any content optimization matters, confirm that AI crawlers are allowed to visit your site. Check your robots.txt file (yourdomain.com/robots.txt) for disallow rules that block GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot. Many hotel CMS platforms and security configurations block these bots by default. A five-minute edit can be the difference between invisible and discoverable.
What to do this week
Disable JavaScript and reload your website. Whatever remains visible is what AI sees. If your room descriptions, amenities, and booking information disappear, you have an immediate problem to fix.
Check your robots.txt. Verify that GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot are not blocked. Ask your web team to add explicit allow rules if needed.
View your page source. Right-click your homepage and key landing pages. Search for your property name, room types, and top amenities. If they're missing from the HTML, prioritize getting them server-rendered.
We ran a test last month that has become standard in every hotel engagement. We took a luxury resort's website, disabled JavaScript in the browser, and reloaded the homepage. The full-screen video disappeared. The room carousel vanished. The booking widget was gone. The restaurant menus, spa services, event spaces, all of it, loaded via JavaScript after the initial page render. What remained was a logo, a navigation bar, and a single paragraph of text that said "Experience luxury redefined."
That stripped-down shell is what GPTBot, ClaudeBot, and PerplexityBot see when they visit. And it's the version that determines whether AI recommends this property or skips it entirely.
The JavaScript blind spot in hospitality
Hotel websites are among the most JavaScript-dependent on the web. Room availability widgets, pricing displays, photo galleries, virtual tours, interactive maps, and booking engines all rely on client-side JavaScript. For human visitors, the experience is polished. For AI crawlers, most of that content doesn't exist.
This isn't theoretical. An analysis of over 500 million GPTBot requests found zero evidence of JavaScript execution, even when the crawler downloaded JS files. AI crawlers fetch raw HTML on the initial request. They don't scroll, click, or wait for API calls. If content isn't in that first response, it's invisible.
The hospitality industry is particularly exposed because the elements AI needs most are often the ones hidden behind JavaScript: room types and descriptions, amenity lists, location details, pricing structures, policies, and FAQ content. These are exactly the facts a traveler asks about when they prompt ChatGPT with "best hotel near downtown Portland with a pool and free parking."
Hospitality consultant Ira Vouk analyzed several major hotel chain websites and found many were blocking most AI bots in their robots.txt files, making them practically invisible on AI platforms. OTAs, by contrast, are wide open to crawlers, with structured, crawlable data that AI can easily parse. This is a significant factor in why OTAs account for 55% of AI hotel citations while hotel websites account for just 13.6%.
What AI needs to find in your HTML
The fix doesn't require a complete website rebuild. It requires ensuring that the information AI needs is present in the server-rendered HTML, regardless of what JavaScript adds after the page loads.
Property fundamentals. Your hotel name, full address, phone number, star rating, and a clear description of what makes the property distinctive should be in plain HTML text on every key page. Not embedded in images. Not loaded via API calls.
Room and amenity details. Room types, bed configurations, key amenities (pool, spa, parking, restaurant, Wi-Fi), and accessibility features need to be in the source code. If a traveler asks AI about your property and the answer is vague or incomplete, this is usually why.
FAQ content. The questions your front desk answers twenty times a day are the same questions travelers ask AI. Check-in time, parking availability, pet policies, distance to the airport, breakfast hours. Build these into a structured FAQ section using real HTML, and mark it up with FAQPage schema so AI models can parse it directly.
Schema markup. Hotel, LocalBusiness, and LodgingBusiness schema types tell AI exactly what kind of entity you are. Add Review schema from your own testimonials. Include geo-coordinates, price ranges, and amenity lists in structured data format. If your schema is injected via Google Tag Manager or client-side JavaScript, AI crawlers won't see it. It must be in the server-rendered HTML.
The robots.txt check most hotels skip
Before any content optimization matters, confirm that AI crawlers are allowed to visit your site. Check your robots.txt file (yourdomain.com/robots.txt) for disallow rules that block GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot. Many hotel CMS platforms and security configurations block these bots by default. A five-minute edit can be the difference between invisible and discoverable.
What to do this week
Disable JavaScript and reload your website. Whatever remains visible is what AI sees. If your room descriptions, amenities, and booking information disappear, you have an immediate problem to fix.
Check your robots.txt. Verify that GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot are not blocked. Ask your web team to add explicit allow rules if needed.
View your page source. Right-click your homepage and key landing pages. Search for your property name, room types, and top amenities. If they're missing from the HTML, prioritize getting them server-rendered.
CONTACT US
