Amazon has announced that starting June 15, 2026, Amazonbot will respect robots.txt directives for controlling crawler access, replacing the previous manual opt-out process. The author notes this is significant because Amazon's aggressive scraping was the original motivation for creating Anubis, a bot-mitigation tool. The author plans to ensure Anubis includes the appropriate robots.txt rules for Amazonbot.
4 Comments
Sort: