The Most Debated Line in Robots.txt
Since OpenAI introduced GPTBot in August 2023, one question has dominated webmaster forums, SEO Twitter, and boardroom conversations: should we block it?
By early 2026, roughly 26% of the top 1,000 websites block GPTBot. The remaining 74% allow it, either intentionally or because they have not updated their robots.txt. But the blocking rate tells you nothing about whether blocking is the right decision for your site.
Let us look at what the data actually shows.
The Case for Blocking GPTBot
Content Protection
If your business depends on proprietary content – original research, paywalled articles, unique product descriptions – allowing GPTBot means your content gets absorbed into OpenAI's training data. Once trained on, your content becomes part of ChatGPT's knowledge base, potentially reducing the incentive for users to visit your site directly.
Publishers with subscription models have the strongest case for blocking. If ChatGPT can answer questions using your journalism, fewer people need to subscribe.
Server Load
GPTBot is not the most aggressive crawler, but it is not gentle either. On mid-size sites, GPTBot typically generates 100-500 requests per day. For sites with limited server resources, this adds up. Blocking eliminates this load entirely.
Philosophical Objection
Some site owners object to their content being used to train commercial AI models without compensation. This is a valid position. Blocking GPTBot is the clearest way to express it.
The Case for Allowing GPTBot
Citation Traffic Is Real
When ChatGPT recommends a product or cites a source, it frequently includes a link. Sites that allow GPTBot and have strong content are seeing measurable referral traffic from chat.openai.com. Early data from botjar customers shows that ecommerce sites with AI-optimized pages receive 3-8% of their organic traffic from AI referrals – and this number is growing month over month.
Blocking GPTBot means zero chance of being cited in ChatGPT responses. Zero referral traffic. Zero brand mentions in the world's most popular AI assistant.
Training Data Influences Recommendations
This is the less obvious but more important point. ChatGPT's recommendations are influenced by its training data. If your product pages are in the training data, ChatGPT "knows" about your products and can recommend them even in conversations that do not trigger web browsing.
If you block GPTBot, your products are excluded from future training runs. Over time, ChatGPT's knowledge of your products becomes stale and eventually nonexistent. Your competitors who allow GPTBot get recommended instead.
The Compounding Effect
AI-powered product discovery is growing exponentially. In 2026, an estimated 30% of product research involves an AI assistant. By 2027, projections suggest 45-50%. Blocking GPTBot today means falling behind in a channel that is compounding rapidly.
What the Data Shows
Analysis of botjar customer data across 200+ ecommerce sites reveals clear patterns:
- Sites that allow GPTBot see an average 12% increase in AI-referred traffic over 6 months
- Sites that block GPTBot see AI referral traffic flatline or decline as cached data ages out
- Sites that allow GPTBot AND optimize for it (clean schema, fast response times, quality content) see 25-40% higher AI referral traffic than sites that simply allow it passively
- No measurable negative SEO impact from allowing GPTBot – Google's rankings are unaffected by your GPTBot policy
A Framework for Your Decision
Rather than a blanket allow or block, consider these factors:
Block GPTBot If:
- Your revenue model depends on content exclusivity (paywalled publications, premium research)
- You have legal or compliance concerns about AI training on your content
- Your content is easily commoditized and you gain no benefit from AI citation
- You have extremely limited server resources and cannot absorb the crawl volume
Allow GPTBot If:
- You sell products and benefit from being recommended in AI conversations
- You want to grow your brand presence in AI-powered search
- Your content is already publicly accessible and indexed by search engines
- You are willing to invest in optimizing how AI crawlers experience your site
Partial Access (Best of Both Worlds)
You do not have to choose all-or-nothing. A selective approach allows GPTBot access to your product pages and public content while blocking sensitive areas:
- Allow: product pages, category pages, blog content, about pages
- Block: checkout flow, account pages, admin areas, internal search results
- Block: thin content pages, paginated results beyond page 2, tag archives
This gives AI crawlers enough data to recommend your products while protecting sensitive and low-value pages. See our complete robots.txt configuration guide for implementation details.
The Bottom Line
For most ecommerce businesses, blocking GPTBot is the wrong call. The citation traffic, brand visibility, and product recommendation benefits outweigh the server load and content reuse concerns. The sites seeing the best results are those that allow GPTBot and actively optimize for it.
But this is not a one-size-fits-all answer. Your decision should be based on your specific business model, content strategy, and competitive landscape – not on what other sites are doing.
See how GPTBot interacts with your site right now. Botjar shows you exactly which pages GPTBot crawls, how often, and whether it can parse your content correctly. Get your free bot audit →