Principal Programmer Jeremy Daer explains how we moved billions of files out of Amazon S3 with zero downtime.

37Signals, now known as Basecamp, shares  insights and practical advice on project management, remote work, collaboration, and productivity. Through blog posts, podcasts, and webinars, 37Signals offers strategies for effective communication, team coordination, and task management in distributed work environments. Readers can learn about project planning techniques, time management strategies, and leadership principles to foster a culture of collaboration and accountability within their teams. Additionally, 37Signals provides case studies, success stories, and real-world examples to inspire organizations to adopt a more flexible and efficient approach to work, driving productivity and innovation.

37signals Dev

37signals moved 5 petabytes (5 billion files) from AWS S3 to self-hosted Pure Storage FlashBlade with zero downtime in under 10 days. The migration required custom Rails tooling for coordination, DuckDB for efficient data partitioning, and careful reconciliation processes. Key challenges included AWS bandwidth limits, the 90-day egress waiver window, and verification anxiety. The team used dual-write strategies, S3 inventory reports, and multiple verification passes before final deletion. The project demonstrated that cloud exit is technically feasible but requires meticulous planning around bandwidth constraints, reconciliation, and psychological readiness for irreversible deletion.

Moving Mountains of Data off S3