Cloudflare crawl endpoint
kept by eddie
Cloudflare launches a managed crawl endpoint that respects robots.txt, aiming to provide a well-behaved alternative to aggressive AI scrapers.
Short for Read Later. Save links from any browser or phone. Click to save, read to clear. The full tour
One line. Many voicesSeek and you shall find
kept by eddie
Cloudflare launches a managed crawl endpoint that respects robots.txt, aiming to provide a well-behaved alternative to aggressive AI scrapers.