Design of a Scraping Platform can be split into 2 categories that require different infrastructure and techniques. The system only has one functional requirement: it must be able to execute arbitrary scrapers on schedule. Scrapyd deploy eggified Python packages Packages may include dependencies in the egg file and different scrapers may use different versions.
1 Comment
Sort: