At Neon, we run hundreds of thousands of Postgres databases as ephemeral Kubernetes pods. Because of our scale-to-zero feature, every time a user connects to their database, there’s a chance that we need to spin up a new Postgres process to serve that connection. The newly spun Postgres instance needs to be configured to serve […]

Neon

Neon deployed NodeLocalDNS across their Kubernetes clusters to optimize DNS performance for hundreds of thousands of ephemeral Postgres databases. By caching DNS requests locally on each node instead of routing them to central CoreDNS pods, they achieved an 84% reduction in 99th percentile latency and 87% improvement in 99.9th percentile latency. The deployment reduced network DNS traffic by 97% (from 2k to 60 requests/s), made traffic scale with nodes rather than pods, and unexpectedly helped identify DNS misconfigurations. The implementation required careful sequencing to avoid race conditions between kube-proxy and the DaemonSet, particularly on nodes with slow iptables rule installation.

Improving DNS performance with NodeLocalDNS