A database permissions change ended up knocking Cloudflare’s proxy offline. Pinpointing the root cause was tricky – but Cloudflare shared a detailed postmortem. Also: announcing The Pragmatic Summit

Pragmatic Engineer is a blog authored by Gergely Orosz, covering topics related to software engineering, career growth, and technology leadership. With a focus on practical advice and real-world experiences, Pragmatic Engineer offers  insights for developers and tech professionals. Developers can learn about software design principles, career advancement strategies, and industry trends through Pragmatic Engineer's articles and essays.

The Pragmatic Engineer

Cloudflare experienced a 6-hour outage affecting major services like ChatGPT, Spotify, and Reddit when a database permissions change in ClickHouse caused the Bot Management module to receive more features than its 200-feature limit, triggering system panics across edge nodes. The root cause took 2.5 hours to identify partly because the status page coincidentally went down, leading engineers to initially suspect a DDoS attack. Cloudflare published a detailed postmortem within 24 hours, revealing how a seemingly innocuous security improvement cascaded into a major incident and highlighting the challenges of maintaining CDN infrastructure that powers much of the internet.

The Pulse: Cloudflare takes down half the internet – but shares a great postmortem