This talk was recorded at NDC Copenhagen in Copenhagen, Denmark. #ndccopenhagen #ndcconferences #developer #softwaredeveloper    

Attend the next NDC conference near you: 
https://ndcconferences.com
https://ndccopenhagen.com/

Subscribe to our YouTube channel and learn every day:   
/        @NDC 

Follow our Social Media!

https://www.facebook.com/ndcconferences
https://twitter.com/NDC_Conferences
https://www.instagram.com/ndc_conferences/

#architecture #continuousdelivery #devops 

No matter how robust your code or how comprehensive your tests, systems will fail—it’s a certainty of software engineering. The key to resilience is not avoiding failure but ensuring rapid detection, effective response, and comprehensive recovery. In this talk, we’ll explore the core principles of on-call rotations and their critical role in handling incidents.
We’ll dive into the lifecycle of an emergency—from detection to restoration—covering essential practices like monitoring, alerting, and ownership. Learn why on-call systems are vital, the dangers of over-relying on senior engineers, and how to empower teams with actionable playbooks, feature flags, rollback strategies, and data restoration techniques. Whether you’re designing your first on-call rotation or optimizing an existing one, this talk provides practical steps to transform chaos into confidence and ensure your business is resilient in the face of failure.

NDC Conferences

A team transformed their incident response from constant system crashes to zero incidents in a year by focusing on four key areas: discovering problems earlier through shift-left testing and production monitoring, analyzing issues with playbooks and proper tooling, recovering through rollbacks instead of forward fixes, and continuously evolving through retros and postmortems. They achieved 99.99% uptime, 20-minute mean time to resolution, and 96% automated problem detection by prioritizing work against KPIs and maintaining comprehensive documentation.

Oops!… We F*ed It Up Again: The Art of Staying Calm When Systems Crash - Alon Kiriati