The Great Bot Blockade: Cloudflare Picks a Side
Cloudflare will now block AI bots from scraping content by default, unless websites explicitly opt in. The company says the move is meant to protect publishers and curb unregulated data harvesting by AI crawlers.
Ryan Browne writes for CNBC
Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow AI crawlers, effectively giving them the ability to prevent bots from scraping data from their websites. The company will also allow publishers to charge AI crawlers for access using a new ‘pay per crawl’ model.
Experts warn the change could hinder AI development and model training if widely adopted.