Things that I Only Learned After Scaling: Non-Obvious Lessons from Production

A collection of 15 hard-won production lessons that only become apparent when systems scale beyond toy workloads. Topics covered include: retries causing thundering herds without backoff and jitter, verbose logging becoming a bottleneck, hidden statefulness in supposedly stateless services, the importance of load balancer connection draining, partial failures being the norm, shared resource contention, misleading metrics without context, tail latency sources like DNS and TLS, feature flags for instant rollback, reversible deploys, DNS as a single point of failure, alerting on symptoms vs root causes, untested failover paths, automating recovery to reduce MTTR, and tech debt multiplying with headcount.