RESOURCES / CULTURAL INSTITUTIONS / WHAT CRAWLERS SEE

What AI crawlers see behind the homepage.

We ran a simple test for a performing arts center in the Midwest. We opened their website, right-clicked "View Page Source," and searched for the name of their current season headliner. Nothing. We searched for their next three events. Nothing. We searched for their address. Also nothing. The entire visible website, the events calendar, the ticket purchasing flow, the artist descriptions, all of it was built with client-side JavaScript. What the browser renders beautifully for a human visitor is a blank page to every AI crawler that matters.


This isn't an edge case. It's the default architecture for most cultural institution websites. And it's the primary reason many museums, historic sites, and performing arts organizations are invisible to ChatGPT, Claude, Perplexity, and Gemini.



AI crawlers don't run JavaScript


This is the single most important technical fact for cultural institutions to understand. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot read raw HTML only. They do not execute JavaScript. When they visit a page, they see exactly what appears in the page source, nothing more.

Googlebot is the exception; it can render JavaScript through a delayed second pass. But AI crawlers skip that entirely. Cloudflare's 2025 analysis found AI-specific bots have grown substantially, with GPTBot among the most active crawlers on the web. Yet their capability is fundamentally different from traditional search crawlers. They are fast, frequent, and completely literal in what they can read.


For museum websites, this creates a specific set of problems because the content most likely to drive AI recommendations is also the content most likely to be JavaScript-dependent.



The five things AI crawlers miss on museum sites


Event calendars. Most institutions use third-party ticketing platforms like Tessitura, Eventbrite, or specialized museum ticketing software. These systems typically inject event listings via JavaScript widgets or iframes. The calendar that visitors see and interact with is invisible to AI crawlers. When someone asks an AI "what's happening at [your museum] this weekend," the AI has no data to work with.


Exhibition pages. Current and upcoming exhibitions are often loaded dynamically, especially when tied to a CMS managing exhibition rotations. If the title, dates, description, and featured artists load via JavaScript, AI crawlers see a template with empty containers.


Collection search results. Collection databases almost universally use search-first architectures where results render client-side. Even individual object pages can be JavaScript-dependent if the CMS delivers content through API calls rather than server-rendered HTML.


Ticketing and pricing information. Hours, admission prices, membership tiers, and ticket availability are frequently managed through external systems that inject data via JavaScript. When AI is asked "how much does it cost to visit [your museum]," it has no answer if pricing lives only in a JavaScript widget.


Interactive maps and visitor information. Campus maps, parking details, accessibility information, and directions commonly live in embedded Google Maps or custom JavaScript. This practical information is what AI surfaces in "planning a visit to..." responses.



The five-minute diagnostic


Three tests reveal what AI crawlers see on your site.


Disable JavaScript in your browser. In Chrome, open DevTools (F12), go to Settings, and check "Disable JavaScript." Reload your homepage, exhibition pages, and event calendar. What remains visible is what AI crawlers see.


View page source. Right-click any page and select "View Page Source." Search for key content: your current exhibition name, your address, your hours, upcoming event titles. If that content isn't in the source, AI crawlers cannot access it.


Check your robots.txt file. Navigate to yourmuseum.org/robots.txt. Look for directives mentioning GPTBot, ClaudeBot, PerplexityBot, or a blanket Disallow: / rule. Paul Calvano's 2025 analysis found nearly 21% of the top 1,000 websites include GPTBot directives. Some museum websites may be blocking AI crawlers without realizing it.



What to fix first


The full technical overhaul is covered in the collection content article. But museums can take immediate steps without a website rebuild.


Add static HTML to your homepage. Your current exhibition names, hours, address, admission pricing, and a brief institutional description should exist as plain text in the page source. If your CMS allows editing the page template, add this content directly. This single change ensures AI has basic facts about your institution.


Create plain HTML event pages. For major exhibitions and signature events, build simple pages with title, dates, description, and visitor information rendered in HTML. Link to the ticketing system for purchases, but ensure informational content exists in crawlable form.


Implement Event and Museum schema markup. JSON-LD in the page head provides AI with structured data about your institution, exhibitions, and events even if visual content loads via JavaScript. Most web teams can implement this without redesigning the site.


Review your robots.txt. Ensure you're not blocking GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot. If your policy allows AI crawling, add explicit Allow rules.


The gap between what visitors see and what AI crawlers see is the most fixable problem in cultural institution AI visibility. The content exists. The authority is real. The barrier is a rendering method never designed with AI discovery in mind.

We ran a simple test for a performing arts center in the Midwest. We opened their website, right-clicked "View Page Source," and searched for the name of their current season headliner. Nothing. We searched for their next three events. Nothing. We searched for their address. Also nothing. The entire visible website, the events calendar, the ticket purchasing flow, the artist descriptions, all of it was built with client-side JavaScript. What the browser renders beautifully for a human visitor is a blank page to every AI crawler that matters.


This isn't an edge case. It's the default architecture for most cultural institution websites. And it's the primary reason many museums, historic sites, and performing arts organizations are invisible to ChatGPT, Claude, Perplexity, and Gemini.



AI crawlers don't run JavaScript


This is the single most important technical fact for cultural institutions to understand. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot read raw HTML only. They do not execute JavaScript. When they visit a page, they see exactly what appears in the page source, nothing more.

Googlebot is the exception; it can render JavaScript through a delayed second pass. But AI crawlers skip that entirely. Cloudflare's 2025 analysis found AI-specific bots have grown substantially, with GPTBot among the most active crawlers on the web. Yet their capability is fundamentally different from traditional search crawlers. They are fast, frequent, and completely literal in what they can read.


For museum websites, this creates a specific set of problems because the content most likely to drive AI recommendations is also the content most likely to be JavaScript-dependent.



The five things AI crawlers miss on museum sites


Event calendars. Most institutions use third-party ticketing platforms like Tessitura, Eventbrite, or specialized museum ticketing software. These systems typically inject event listings via JavaScript widgets or iframes. The calendar that visitors see and interact with is invisible to AI crawlers. When someone asks an AI "what's happening at [your museum] this weekend," the AI has no data to work with.


Exhibition pages. Current and upcoming exhibitions are often loaded dynamically, especially when tied to a CMS managing exhibition rotations. If the title, dates, description, and featured artists load via JavaScript, AI crawlers see a template with empty containers.


Collection search results. Collection databases almost universally use search-first architectures where results render client-side. Even individual object pages can be JavaScript-dependent if the CMS delivers content through API calls rather than server-rendered HTML.


Ticketing and pricing information. Hours, admission prices, membership tiers, and ticket availability are frequently managed through external systems that inject data via JavaScript. When AI is asked "how much does it cost to visit [your museum]," it has no answer if pricing lives only in a JavaScript widget.


Interactive maps and visitor information. Campus maps, parking details, accessibility information, and directions commonly live in embedded Google Maps or custom JavaScript. This practical information is what AI surfaces in "planning a visit to..." responses.



The five-minute diagnostic


Three tests reveal what AI crawlers see on your site.


Disable JavaScript in your browser. In Chrome, open DevTools (F12), go to Settings, and check "Disable JavaScript." Reload your homepage, exhibition pages, and event calendar. What remains visible is what AI crawlers see.


View page source. Right-click any page and select "View Page Source." Search for key content: your current exhibition name, your address, your hours, upcoming event titles. If that content isn't in the source, AI crawlers cannot access it.


Check your robots.txt file. Navigate to yourmuseum.org/robots.txt. Look for directives mentioning GPTBot, ClaudeBot, PerplexityBot, or a blanket Disallow: / rule. Paul Calvano's 2025 analysis found nearly 21% of the top 1,000 websites include GPTBot directives. Some museum websites may be blocking AI crawlers without realizing it.



What to fix first


The full technical overhaul is covered in the collection content article. But museums can take immediate steps without a website rebuild.


Add static HTML to your homepage. Your current exhibition names, hours, address, admission pricing, and a brief institutional description should exist as plain text in the page source. If your CMS allows editing the page template, add this content directly. This single change ensures AI has basic facts about your institution.


Create plain HTML event pages. For major exhibitions and signature events, build simple pages with title, dates, description, and visitor information rendered in HTML. Link to the ticketing system for purchases, but ensure informational content exists in crawlable form.


Implement Event and Museum schema markup. JSON-LD in the page head provides AI with structured data about your institution, exhibitions, and events even if visual content loads via JavaScript. Most web teams can implement this without redesigning the site.


Review your robots.txt. Ensure you're not blocking GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot. If your policy allows AI crawling, add explicit Allow rules.


The gap between what visitors see and what AI crawlers see is the most fixable problem in cultural institution AI visibility. The content exists. The authority is real. The barrier is a rendering method never designed with AI discovery in mind.

CONTACT US