Cloudflare crawl endpoint

kept by eddie mar 15

Cloudflare launches a managed crawl endpoint that respects robots.txt, aiming to provide a well-behaved alternative to aggressive AI scrapers.

Full-Text Search with DuckDB
DuckDB's full-text search extension offers BM25 scoring, stemming, and stop-word removal but lacks result highlighting and phrase queries.
also on Beautiful Soup, #developer-tools
GitHub - firecrawl/firecrawl-workflows
Firecrawl Workflows provides outcome-focused skills for AI coding agents to produce research reports, SEO audits, lead lists, and other deliverables using the Firecrawl web extraction tool.
also on Firecrawl, #automation, #developer-tools
CLI | Firecrawl
The Firecrawl CLI tool lets AI agents and developers scrape, crawl, search, map, monitor, and interact with web pages directly from the terminal.
also on Firecrawl, #developer-tools
Browser Run: give your agents a browser
Cloudflare renames Browser Rendering to Browser Run, adding Live View, Human in the Loop, CDP access, WebMCP support, session recordings, and 4x higher concurrency for AI agents.
also on Playwright, Cloudflare, #ai, #developer-tools
How we cut build times by two-thirds by deleting our CMS
Sentry cut build times from 14 to under 4 minutes by replacing its headless CMS with Astro, Markdown files, and Claude Code skills for content management.
also on Playwright, #web-development, #developer-tools