Anthropic and OpenAI are crawling the web even more and not giving much back
A venomous Phormictopus antillensis spider crawls on a keyboard
Reuters
- Cloudflare data shows Anthropic and OpenAI are crawling the web and sending very few referrals.
- The crawl-to-refer ratio has deteriorated compared to early September.
- The data suggests AI companies are taking more than they give back to the web.
Are AI giants nurturing the web, the most valuable store of human data the world will ever see? Or are they scraping content for free and giving little back?