Design of a Scraping Platform can be split into 2 categories that require different infrastructure and techniques. The system only has one functional requirement: it must be able to execute arbitrary scrapers on schedule. Scrapyd deploy eggified Python packages Packages may include dependencies in the egg file and different scrapers may use different versions.

1m read timeFrom hackernoon.com
Post cover image
1 Comment

Sort: