Networking is a black box, we used eBPF to open it

Railway's network engineer describes building Network Flows, a real-time network observability feature using eBPF to capture packet-level data from containers. The system enriches raw socket buffers with service context, batches 10,000 flows per second per host, and writes roughly one million rows per second to ClickHouse. A key architectural challenge was reducing ClickHouse merge pressure by consolidating ~1,000 independent host writers into three ingestion pipeline writers. The resulting UI visualizes traffic as animated pipes on Railway's service canvas, with pipe width representing throughput and clickable logs exposing kernel-level drop codes. The motivation was twofold: reduce support ticket burden caused by customers misdiagnosing network issues as application bugs, and give developers a top-down view of network problems without requiring tcpdump expertise.