Reading up on web-scraping

Reddit blocks automated access from the scraping tool or script used to fetch this SaaS-related post, returning a network security block page.

Parallel releases a CLI tool that gives terminal-based AI agents access to web search, clean content extraction, deep research, and structured data enrichment.

The Firecrawl CLI tool lets AI agents and developers scrape, crawl, search, map, monitor, and interact with web pages directly from the terminal.

Cloudflare's Browser Rendering adds a /crawl endpoint that lets developers crawl entire websites via a single API call, outputting HTML, Markdown, or JSON.

Tadpole is a KDL-based DSL for web scraping that enables modular, human-behavior-simulating scraper logic importable directly from Git without package managers.

Yandori News Flow uses HNSW clustering to track how stories spread across 200k websites in real time, showing propagation timelines.