Imagine spending thousands of dollars on content marketing, only to lock the door when the most important readers try to visit. That is exactly what is happening to thousands of businesses right now. In an effort to protect their data or simply due to outdated server configurations, companies are accidentally blocking AI crawlers from accessing their websites.
If ChatGPT, Perplexity, and Claude cannot read your site, they cannot cite your brand. You are effectively invisible in the era of AI search. Here is how to check if you are blocking the bots, and how to fix it.
The Great Bot Blockade
In 2023 and 2024, there was a massive wave of publishers and businesses updating their robots.txt files to block AI companies from scraping their content for training data.
At the time, this made sense. But the landscape has shifted. AI is no longer just a training mechanism; it is a primary search and discovery engine. By blocking AI bots today, you aren't just protecting your IP—you are blocking potential customers who use AI to find products and services.
How AI Crawlers Work
When a user asks Perplexity or ChatGPT a question that requires real-time information, the AI engine deploys a crawler to scan the live web for answers.
These crawlers identify themselves using specific User-Agents. The most common ones you need to know are:
GPTBot(OpenAI / ChatGPT)OAI-SearchBot(OpenAI's dedicated search crawler)ClaudeBot(Anthropic)PerplexityBot(Perplexity AI)Google-Extended(Google's AI features)
When these bots hit your server, the first thing they check is your robots.txt file. If they see a "Disallow" directive, they turn around and leave. The AI engine will then formulate its answer using your competitors' websites instead.
How to Check Your robots.txt File
Checking your configuration is simple.
- Open your web browser.
- Type your domain name followed by
/robots.txt(e.g.,https://yourdomain.com/robots.txt). - Look for the User-Agents listed above.
If you see something like this, you are blocking OpenAI from reading your site:
User-agent: GPTBot
Disallow: /
To allow AI crawlers, you either need to remove the specific Disallow directives for these bots, or explicitly Allow them if your default setting blocks all unknown agents.
The Hidden Culprit: Web Application Firewalls (WAF)
Sometimes, your robots.txt file looks perfectly fine, but AI crawlers are still failing to reach your site. The culprit is often your Web Application Firewall (WAF), such as Cloudflare or AWS Shield.
Many WAFs have "Bot Fight Mode" or automated security rules that aggressively block anything they don't recognize as a standard human browser or a legacy Google bot. You must review your server-level security settings and ensure that the IP ranges and User-Agents for major AI platforms are whitelisted.
Don't Guess. Test It.
Technical SEO for AI is unforgiving. A single misconfigured line of text can erase your brand from millions of AI-generated answers.
You don't have to guess if your site is accessible. Run our free tool to see if AI crawlers can reach your homepage. Get your free snapshot today and ensure your technical foundation is ready for AI search.