Breaking News
Cloudflare has rolled out two new tools to help website owners easily manage access to their content by AI bots, particularly those that scrape data for training large language models. The company now offers a simple way for customers to let Cloudflare automatically create and manage a robots.txt file, specifying whether or not AI crawlers can access their websites. Additionally, Cloudflare users can now choose to block AI bots specifically from pages that generate revenue through advertising, protecting the financial interests of content creators.
For years, website owners welcomed traditional search engine crawlers like Googlebot because they drove traffic to their sites, increasing visibility and ad revenue. However, the landscape is shifting. A new wave of AI crawlers has emerged—bots that gather data from websites to train artificial intelligence systems. Unlike search crawlers, these AI bots offer little in return. They rarely direct users back to the original sites, making the exchange of data one-sided and exploitative.
Cloudflare’s Radar team analyzed traffic patterns and discovered a sharp imbalance. In June 2025, Google crawled websites about 14 times for every one visitor it referred back. In stark contrast, OpenAI’s crawl-to-referral ratio stood at a staggering 1,700:1, while Anthropic’s was an even more extreme 73,000:1. This data highlights how AI bots are consuming vast amounts of content without contributing to publisher traffic, undermining the long-standing “crawl-for-traffic” relationship.
Despite the risks, many websites remain unprotected. Only 37% of the top 10,000 domains use a robots.txt file, indicating how underutilized this basic defense remains. Cloudflare aims to change that. In 2024, it offered a free one-click feature to block AI scrapers, which over a million users have activated. As a result, traffic from bots like Bytespider dropped by over 71%, though others like GPTBot saw significant growth.
With its latest updates, Cloudflare is reinforcing its commitment to an open, fair internet—one where independent publishers can thrive without having their content unfairly exploited by AI companies.
For years, website owners welcomed traditional search engine crawlers like Googlebot because they drove traffic to their sites, increasing visibility and ad revenue. However, the landscape is shifting. A new wave of AI crawlers has emerged—bots that gather data from websites to train artificial intelligence systems. Unlike search crawlers, these AI bots offer little in return. They rarely direct users back to the original sites, making the exchange of data one-sided and exploitative.
Cloudflare’s Radar team analyzed traffic patterns and discovered a sharp imbalance. In June 2025, Google crawled websites about 14 times for every one visitor it referred back. In stark contrast, OpenAI’s crawl-to-referral ratio stood at a staggering 1,700:1, while Anthropic’s was an even more extreme 73,000:1. This data highlights how AI bots are consuming vast amounts of content without contributing to publisher traffic, undermining the long-standing “crawl-for-traffic” relationship.
Despite the risks, many websites remain unprotected. Only 37% of the top 10,000 domains use a robots.txt file, indicating how underutilized this basic defense remains. Cloudflare aims to change that. In 2024, it offered a free one-click feature to block AI scrapers, which over a million users have activated. As a result, traffic from bots like Bytespider dropped by over 71%, though others like GPTBot saw significant growth.
With its latest updates, Cloudflare is reinforcing its commitment to an open, fair internet—one where independent publishers can thrive without having their content unfairly exploited by AI companies.
See What’s Next in Tech With the Fast Forward Newsletter
Tweets From @varindiamag
Nothing to see here - yet
When they Tweet, their Tweets will show up here.
