spatie/crawler is a PHP package for crawling websites concurrently using Guzzle promises. Version 9 introduces a typed CrawlResponse object, closure callbacks and observer classes for crawl events, scope controls (internal-only, depth limiting), URL collection, throttling strategies (fixed and adaptive), automatic retries, streaming support, a FinishReason enum, and a fake() method for testing without real HTTP requests. JavaScript rendering is also supported via a CloudflareRenderer or spatie/browsershot.
Table of contents
# Handling Crawl Events# CrawlResponse# Collecting URLs and Controlling Scope# Testing with fake()# Other HighlightsSort: