Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. https://blog.cloudflare.com/introducing-pay-per-crawl/

Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf. https://blog.cloudflare.com/control-content-use-for-ai-training/

From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/